* upgrade vagrant box to xenial

this allows the redis version to be upgraded to support the new redis
features used in the activity tracker

* add libpam0g package to vagrant box

this is required for native extensions of gems to build after the
addition of PAM support was added in #5303

RVM has a known issue with installing Ruby 2.5 on the version of Ubuntu
the Vagrant box is using: https://github.com/rvm/rvm/issues/4291

This bug was preventing any gem installs in the vagrant box

* Specs for language detection

* Use CharlockHolmes instead of NKF

* Correct mistakes

* Correct style

* Set hint_enc instead of falling back and strip_tags

* Improve specs

* Add dependencies

as mentioned by ElvenSpellmaker here: https://github.com/rvm/rvm/issues/4068

Adds a workaround for the issue mentioned by @abcang here: https://github.com/tootsuite/mastodon/pull/3897#issuecomment-310436668 and makes sure that we're using the stable installer to install the stable version.

Compact Language Detector v3 (CLD3) is the successor of CLD2, which was
used in the previous implementation. CLD3 includes improvements since CLD2,
and supports newer compilers. On the other hand, it has additional
requirements and cld3-ruby, the FFI of CLD3 for Ruby, is still new and may
be still inmature.

Though CLD3 is named after CLD2, it is implemented with a neural network
model, different from the old implementation, which is based on a Naïve
Bayesian classifier.

CLD3 supports newer compilers, such as GCC 6. CLD2 is not compatible with
GCC 6 because it assigns negative values to varibales typed unsigned.
(see internal/cld_generated_cjk_uni_prop_80.cc) The support for GCC 6 and
newer compilers are essential today, when some server operating system
such as Ubuntu Server 16.10 has GCC 6 by default.

On the one hand, CLD3 requires C++11 support. Environments with old
compilers such as Ubuntu Server 14.04 needs to update the system or install
a newer compiler.

CLD3 needs protocol buffers as a new dependency. However,it is not
considered problematic because major server operating systems, CentOS and
Ubuntu Server provide them.

The FFI cld3-ruby was written by me (Akihiko Odaki) for use in Mastodon.
It is still new and may be inmature, but confirmed to pass existing tests.

* changed to run foreman with vagrant

* change port specification method and database setting

* delete changes of environment variable related code

* hard coated the port

Export environment variables from .env.vagrant before running rails commands in Vagrant provisioning (#2343)

Fix #1940

Prior to this commit, the vagrantfile installs ruby 2.3.3. Due to commit mastodon@7951e7f, this no longer works.

Change the provision scrip so that 2.3.1 is installed via rbenv. Note this drastically increases provision time.

The current Vagrantfile fails to build upon provisioning, as it lacks yarn.

Change the Vagrantfile to add the yarn repository, and install yarn.