* Add entity cache

Use a caching layer for mentions and custom emojis that are
dynamically extracted from text.

Reduce duplicate text extractions

* Fix code style issue

The goal here isn't to prevent these hashtags from existing, but just to strongly curtail their usage; The hashtags may still exist in the database via federated status, or from being created prior to this feature.

* Update annotate to version 2.7.3

* Update aws-sdk-s3 to version 1.9.2

* Update browser to version 2.5.3

* Update capistrano to version 3.10.2

* Update domain_name to version 0.5.20180417

* Update http to version 3.2.0

* Update lograge to version 0.10.0

* Update oj to version 3.5.1

* Update parallel_tests to version 2.21.3

* Update puma to version 3.11.4

* Update rubocop to version 0.55.0

* Update scss_lint to version 0.57.0

* Update simplecov to version 0.16.1

* Update tty-command to version 0.8.0

* Update tty-prompt to version 0.16.0

* Update pkg-config to version 1.3.0

* Update fog-local to version 0.5.0

* Update fog-openstack to version 0.1.25

* Update devise-two-factor to version 3.0.3

* bundle update

* Added a timeline for Direct statuses
* Lists all Direct statuses you've sent and received
* Displayed in Getting Started
* Streaming server support for direct TL

* Changes to match other timelines in 2.0

* Fix #201: Account archive download

* Export actor and private key in the archive

* Optimize BackupService

- Add conversation to cached associations of status, because
  somehow it was forgotten and is source of N+1 queries
- Explicitly call GC between batches of records being fetched
  (Model class allocations are the worst offender)
- Stream media files into the tar in 1MB chunks
  (Do not allocate media file (up to 8MB) as string into memory)
- Use #bytesize instead of #size to calculate file size for JSON
  (Fix FileOverflow error)
- Segment media into subfolders by status ID because apparently
  GIF-to-MP4 media are all named "media.mp4" for some reason

* Keep uniquely generated filename in Paperclip::GifTranscoder

* Ensure dumped files do not overwrite each other by maintaing directory partitions

* Give tar archives a good name

* Add scheduler to remove week-old backups

* Fix code style issue

Time window of 6 hours

* Add full-text search for authorized statuses

- Search API will return statuses that match the query
- Only for logged in users
- Only if you are author of the status,
- Or you were mentioned in it
- Or you favourited or reblogged it
- Configuration over `ES_ENABLED`, `ES_HOST`, `ES_PORT`, `ES_PREFIX`
- Run `rails chewy:deploy` to create & populate index

Fix #5880
Fix #4293
Fix #1152

* Add commented out docker-compose configuration for ES container

* Optimize index import, filter search results

* Add basic normalization to the index

* Add better stemming and normalization to the index

* Skip webfinger request if search query includes both @ and a space

* Fix code style

* Visually separate search result sections

* Fix code style issues

* Add GET /api/v1/instance/peers API to reveal known domains

* Add GET /api/v1/instance/activity API

* Make new APIs disableable, exclude private statuses from activity stats

* Fix code style issue

* Fix week timestamps

this should fix *all* remaining visibility-related mastodon ostatus infoleaks.
thanks to @csaurus@gnusocial.de for pointing out the infoleak.

* Add structure for lists

* Add list timeline streaming API

* Add list APIs, bind list-account relation to follow relation

* Add API for adding/removing accounts from lists

* Add pagination to lists API

* Add pagination to list accounts API

* Adjust scopes for new APIs

- Creating and modifying lists merely requires "write" scope
- Fetching information about lists merely requires "read" scope

* Add test for wrong user context on list timeline

* Clean up tests

All the migrations have been updated to use BIGINTs for ID fields in the DB, but ActiveRecord needs to be told to treat those values as BIGINT as well. This PR does that.

The main change of this PR is removing `order by visibility` hack.

This was introduced to force using of `index_statuses_on_account_id` instead of PK index, but it seems no longer needed probably due to `index_statuses_on_account_id_id`. Removing this avoids reading all rows, so really improves first fetching of the user who has lot of statuses.

I have also changed JOIN to IN + subquery, which slightly faster in most cases.

- For some reason, :if option on before_action did not work. It got
  executed every time, returned false, and the action run anyway,
  which led to the current_sign_in_at and sign_in_count being
  updated on every request
- Return "do not filter" early in FeedManager#filter_from_home? if
  the status is authored by receiver. Usually this method is not
  called for own statuses at all, but it is called when Feed#get
  uses the database
- Return early if #reload_stale_associations! has nothing to load
  to save a database query with WHERE 1=0

* Retoot count increases without reason

-The store_uri method for Statuses was being called on after_create and causing reblogs to be incremented twice.
-This calls it when the transaction is finished by using after_create_commit.
-Fixes #4916.

* Added test case for after_create_commit callback for checking reblog count.

* Rewrote test to keep original, but added one for only the after_create_commit callback.

- Rename Mastodon::TimestampIds into Mastodon::Snowflake for clarity
- Skip for statuses coming from inbox, aka delivered in real-time
- Skip for statuses that claim to be from the future

* Add emoji autosuggest

Some credit goes to glitch-soc/mastodon#149

* Remove server-side shortcode->unicode conversion

* Insert shortcode when suggestion is custom emoji

* Remove remnant of server-side emojis

* Update style of autosuggestions

* Fix wrong emoji filenames generated in autosuggest item

* Do not lazy load emoji picker, as that no longer works

* Fix custom emoji autosuggest

* Fix multiple "Custom" categories getting added to emoji index, only add once

* Custom emoji

- In OStatus: `<link rel="emoji" name="coolcat" href="http://..." />`
- In ActivityPub: `{ type: "Emoji", name: ":coolcat:", href: "http://..." }`
- In REST API: Status object includes `emojis` array (`shortcode`, `url`)
- Domain blocks with reject media stop emojis
- Emoji file up to 50KB
- Web UI handles custom emojis
- Static pages render custom emojis as `<img />` tags

Side effects:

- Undo #4500 optimization, as I needed to modify it to restore
  shortcode handling in emojify()
- Formatter#plaintext should now make sure stripped out line-breaks
  and paragraphs are replaced with newlines

* Fix emoji at the start not being converted

* Decouple Status#local? from uri being nil

* Replace on-the-fly URI generation with stored URIs

- Generate URI in after_save hook for local statuses
- Use static value in TagManager when available, fallback to tag format
- Make TagManager use ActivityPub::TagManager to understand new format
- Adjust tests

* Use other heuristic for locality of old statuses, do not perform long query

* Exclude tombstone stream entries from Atom feed

* Prevent nil statuses from landing in Pubsubhubbub::DistributionWorker

* Fix URI not being saved (#4818)

* Add more specs for Status

* Save generated uri immediately

and also fix method order to minimize diff.

* Fix alternate HTML URL in Atom

* Fix tests

* Remove not-null constraint from statuses migration to speed it up

* Make PreviewCard records reuseable between statuses

**Warning!** Migration truncates preview_cards tablec

* Allow a wider thumbnail for link preview, display it in horizontal layout (#4648)

* Delete preview cards files before truncating

* Rename old table instead of truncating it

* Add mastodon:maintenance:remove_deprecated_preview_cards

* Ignore deprecated_preview_cards in schema definition

* Fix null behaviour

Requires moving Atom rendering from DistributionWorker (where
`stream_entry.status` is already nil) to inline (where
`stream_entry.status.destroyed?` is true) and distributing that.

Unfortunately, such XML renderings can no longer be easily chained
together into one payload of n items.

* Pinned statuses

* yarn manage:translations

- Use unicode when selecting emoji through picker
- Convert shortcodes to unicode when storing text input server-side
- Do not convert shortcodes in JS anymore

* Fix boolean columns sometimes having a null value

* Fix wrong value being set instead of null

* Fix regression from #3842

Simplify the query by omitting all direct statuses. Private statuses
are allowed because they are from accounts we are following (so
by definition)

Resolves #3887 (alternative)

* Adjust test

The classes using Status.as_home_timeline, namely Feed and
PrecomputeFeedService are expected to filter direct statuses as
FanOutWriteService does, but their filtering were incomplete or missing.

This commit solves the problem by filtering direct statuses in
as_home_timeline as the other similar methods such as as_public_timeline
does.

* Default to nil for statuses.language

* Language detection defaults to nil instead of instance UI default

* Add a StatusFilter class to identify visibility of statuses by accounts

* Extract StatusThreadingConcern from Status

* Clarify purpose of checking for nil account

* Remove trailing whitespace in i18n mailers

* Use query methods instead of #present? on AR attributes

* Delegate Status#account_domain method

* Delegate Mention #account_username and #account_acct methods