docs: add note about maintenance status (#153)

* Update getting-started.md

* Update README.md

* Apply suggestions from code review

Co-authored-by: Sarah Dayan <sarah.dayan@algolia.com>

Co-authored-by: Sarah Dayan <sarah.dayan@algolia.com>

.gitignore

@@ -5,7 +5,10 @@ coverage/
 docs-dev/
 gemfiles/*.lock
 spec/site/_site
+spec/integration/site/_site
 
+spec/site/.jekyll-cache/
+.jekyll-cache/
 
 .DS_Store
 .*.sw[a-z]

.travis.yml

@@ -3,9 +3,9 @@ cache: bundler
 before_script: bundle update
 script: bundle exec rake test
 rvm:
+ - 2.6.2
  - 2.4.2
  - 2.3.5
- - 2.3.0
 notifications:
   email:
     on_success: never

CONTRIBUTING.md

@@ -38,6 +38,16 @@ group :jekyll_plugins do
 end
 ```
 
+## Running integration tests
+
+Integration tests will do a full jekyll run, and push data to an Algolia index,
+checking that records and settings are correctly saved. It is the slowest
+possible kind of tests, but also the one closest to a real implementation.
+
+Running those tests requires a real Algolia plan. You need to define
+`ALGOLIA_APPLICATION_ID`, `ALGOLIA_API_KEY` and `ALGOLIA_INDEX_NAME` (we suggest
+using `direnv` for that), and then run `./scripts/test_integration`.
+
 ## Linting
 
 Run `rake lint` to check the style of all ruby files. Run `rake

README.md

@@ -1,15 +1,16 @@
 # Jekyll Algolia Plugin
 
-[![gem version][1]](https://rubygems.org/gems/jekyll-algolia)
+[![gem version][1]][16]
 ![ruby][2]
 ![jekyll][3]
-[![build master][4]](https://travis-ci.org/algolia/jekyll-algolia)
-[![coverage master][5]](https://coveralls.io/github/algolia/jekyll-algolia?branch=master)
-[![build develop][6]](https://travis-ci.org/algolia/jekyll-algolia)
-[![coverage develop][7]](https://coveralls.io/github/algolia/jekyll-algolia?branch=develop)
+[![build master][4]][17]
+[![build develop][6]][17]
+[![coverage master][5]][18]
 
 Add fast and relevant search to your Jekyll site.
 
+> While this plugin was created by Algolia, it is not an officially supported API client. It is possible that future major versions of Jekyll break compatibility, or require changes.
+
 ## Usage
 
 ```shell
@@ -21,7 +22,7 @@ This will push the content of your Jekyll website to your Algolia index.
 ## Documentation
 
 Full documentation can be found on
-[https://community.algolia.com/jekyll-algolia/](https://community.algolia.com/jekyll-algolia/getting-started.html)
+[https://community.algolia.com/jekyll-algolia/][20]
 
 ## Installation
 
@@ -42,15 +43,15 @@ Once this is done, download all dependencies with `bundle install`.
 
 ## Basic configuration
 
-You need to provide certain Algolia credentials for this plugin to *index* your
+You need to provide certain Algolia credentials for this plugin to _index_ your
 site.
 
-*If you don't yet have an Algolia account, you can open a free [Community plan
-here][8]. Once signed in, you can get your
-credentials from [your dashboard][9].*
+_If you don't yet have an Algolia account, we suggest that you open a free
+[Community plan here][8]. You can find more information about the Algolia plans
+[in our FAQ][10]._
 
-Once you have your credentials, you should define your `application_id` inside
-your `_config.yml` file like this:
+Once signed in, you should get your application ID from [your dashboard][9] and
+define it inside your `_config.yml` file like this:
 
 ```yaml
 # _config.yml
@@ -72,32 +73,59 @@ Note that `ALGOLIA_API_KEY` should be set to your admin API key.
 
 ## More about the Community plan
 
-The Algolia [Community plan][10] lets you host up to 10k records and perform up to
-100k add/edit/delete operations per month (search operations are free). The plan
-is entirely free, with no time limit.
+The Algolia [Community plan][11] lets you host up to 10k records and perform up
+to 100k add/edit/delete operations per month (search operations are free). The
+plan is entirely free, with no time limit.
 
 What we ask in exchange is that you display a "Search by Algolia" logo next to
-your search results. Our [InstantSearch libraries][11] have a simple boolean
+your search results. Our [InstantSearch libraries][12] have a simple boolean
 option to toggle that on an off. If you want more flexibility, you can find
-all versions of our logo [here][12].
+all versions of our logo [here][13].
+
+If you need more information about the other Algolia plans, you can [check our
+FAQ][10].
 
 # Thanks
 
-Thanks to [Anatoliy Yastreb][13] for a [great tutorial][14] on creating Jekyll
+Thanks to [Anatoliy Yastreb][14] for a [great tutorial][15] on creating Jekyll
 plugins.
 
-
 [1]: https://badge.fury.io/rb/jekyll-algolia.svg
+
 [2]: https://img.shields.io/badge/ruby-%3E%3D%202.3.0-green.svg
+
 [3]: https://img.shields.io/badge/jekyll-%3E%3D%203.6.0-green.svg
+
 [4]: https://img.shields.io/badge/dynamic/json.svg?label=build%3Amaster&query=value&uri=https%3A%2F%2Fimg.shields.io%2Ftravis%2Falgolia%2Fjekyll-algolia.json%3Fbranch%3Dmaster
-[5]: https://img.shields.io/badge/dynamic/json.svg?label=coverage%3Amaster&colorB=&prefix=&suffix=%25&query=$.coverage_change&uri=https%3A%2F%2Fcoveralls.io%2Fgithub%2Falgolia%2Fjekyll-algolia.json%3Fbranch%3Dmaster
+
+[5]: https://coveralls.io/repos/github/algolia/jekyll-algolia/badge.svg?branch=master
+
 [6]: https://img.shields.io/badge/dynamic/json.svg?label=build%3Adevelop&query=value&uri=https%3A%2F%2Fimg.shields.io%2Ftravis%2Falgolia%2Fjekyll-algolia.json%3Fbranch%3Ddevelop
-[7]: https://img.shields.io/badge/dynamic/json.svg?label=coverage%3Adevelop&colorB=&prefix=&suffix=%25&query=$.coverage_change&uri=https%3A%2F%2Fcoveralls.io%2Fgithub%2Falgolia%2Fjekyll-algolia.json%3Fbranch%3Ddevelop
+
+[7]: https://coveralls.io/repos/github/algolia/jekyll-algolia/badge.svg?branch=develop
+
 [8]: #more-about-the-community-plan
+
 [9]: https://www.algolia.com/api-keys
-[10]: https://www.algolia.com/users/sign_up/hacker
-[11]: https://community.algolia.com/instantsearch.js/
-[12]: https://www.algolia.com/press#resources
-[13]: https://github.com/ayastreb/
-[14]: https://ayastreb.me/writing-a-jekyll-plugin/
+
+[10]: https://community.algolia.com/jekyll-algolia/faq.html#how-many-records-will-the-plugin-need
+
+[11]: https://www.algolia.com/users/sign_up/hacker
+
+[12]: https://community.algolia.com/instantsearch.js/
+
+[13]: https://www.algolia.com/press/?section=brand-guidelines
+
+[14]: https://github.com/ayastreb/
+
+[15]: https://ayastreb.me/writing-a-jekyll-plugin/
+
+[16]: https://rubygems.org/gems/jekyll-algolia
+
+[17]: https://travis-ci.org/algolia/jekyll-algolia
+
+[18]: https://coveralls.io/github/algolia/jekyll-algolia?branch=master
+
+[19]: https://coveralls.io/github/algolia/jekyll-algolia?branch=develop
+
+[20]: https://community.algolia.com/jekyll-algolia/getting-started.html

docs-src/config.js

@@ -27,6 +27,7 @@ const sidebarMenu = [
     items: [
       { title: 'Getting Started', url: 'getting-started.html' },
       { title: 'How it works', url: 'how-it-works.html' },
+      { title: 'FAQ', url: 'faq.html' },
     ],
   },
   {

docs-src/package.json

@@ -35,7 +35,7 @@
     "hasha": "3.0.0",
     "http-server": "0.10.0",
     "jstransformer-markdown-it": "2.0.0",
-    "lodash": "4.17.4",
+    "lodash": "4.17.5",
     "markdown-it": "8.4.0",
     "markdown-it-anchor": "4.0.0",
     "metalsmith": "2.3.0",
@@ -45,7 +45,7 @@
     "metalsmith-sass": "1.4.0",
     "ms-webpack": "2.0.0",
     "node-sass": "4.7.2",
-    "nodemon": "1.12.5",
+    "nodemon": "1.18.7",
     "postcss": "6.0.14",
     "postcss-loader": "2.0.9",
     "postcss-scss": "1.0.2",
@@ -66,8 +66,8 @@
     "webpack-hot-middleware": "2.21.0"
   },
   "engines": {
-    "node": "^9.2.0",
-    "yarn": "^1.3.2"
+    "node": ">=9.2.0",
+    "yarn": ">=1.3.2"
   },
   "renovate": {
     "extends": [

docs-src/src/faq.md

@@ -0,0 +1,100 @@
+---
+title: Frequently Asked Questions
+layout: content-with-menu.pug
+---
+
+# Frequently Asked Questions
+
+## How many records will the plugin need?
+
+The plugin will not create an exact mapping of `1 page = 1 record`. Instead, it
+will split all pages into smaller chunks, and each chunk will be saved as a
+record. Splitting into small chunks is key to offer the best relevance of
+results.
+
+The default chunking mechanism is to create one chunk per paragraph of content.
+So, for a blog post that is made up of 10 paragraphs of text, it will create 10
+records.
+
+Estimating the number of records you will need can be tricky as it depends on
+both the number of pages you have, and on the average length of them. Some
+configuration options (such as [nodes_to_index][1]) can also influence the final
+result.
+
+The following table should give you a ballpark estimate of what to expect. All
+calculations were done with an average of **15 paragraphs per page**, on a
+timeline of **one year**.
+
+| update frequency            | # of new pages | # of records | Algolia plan   |
+| --------------------------- | -------------- | ------------ | -------------- |
+| ~1 new page per week        | ~50            | ~750         | [Community][2] |
+| ~1 new page per day         | ~400           | ~6.000       | [Community][3] |
+| ~2 new pages per day        | ~700           | ~10.500      | [Essential][4] |
+| ~1 new page per hour        | ~8800          | ~132.000     | [Essential][5] |
+| ~1 new page every 5 minutes | ~105.000       | ~1.575.000   | [Plus][6]      |
+| More?                       | More?          | More?        | [Business][7]  |
+
+## One of my records is too big. What can I do?
+
+If you get an error about one of your records being too big, be sure to update
+the plugin to the latest version. We keep improving the plugin with ways of
+making the records smaller.
+
+If you're still having an error, you should check the `.json` log file that has
+been created in your source directory. This will contain the content of the
+record that has been rejected. It might give you hints about what went wrong.
+
+A common cause for this issue often lies in the page HTML. Maybe the HTML is
+malformed (a tag is not closed for example), or instead of several small
+paragraphs there is only one large paragraph. This can cause the parser to take
+the whole page content (instead of small chunks of it) to create the records.
+
+If you don't find where the issue is coming from, feel free to open a [GitHub
+issue][8] with a copy of the log file and we'll be happy to help you.
+
+## How can I tweak the relevance ranking?
+
+The plugin default configuration will rank results based on their textual
+relevance. You can adapt the ranking to fit your needs by using a
+combination of front-matter and Algolia index settings.
+
+For example, if you know some of your blog posts are popular, you might want to
+give them a boost in their ranking. To do so, add a `popular: true` entry to the
+front-matter of such posts. Any custom key added to a front-matter is
+automatically pushed to their corresponding records.
+
+Then, you would have to edit your `_config.yml` file to pass a custom
+`settings.customRanking` value. The `customRanking` is one of the way ranking
+can be configured in an Algolia index and follows a tie-breaking algorithm. You
+can find more information about the way it works either in the [official
+documentation][9] or in [this video][10].
+
+The default `customRanking` used by the plugin is [defined here][11] and use the
+date, weight of the header and position in the page by default. You can
+overwrite it to also take the `popular` flag into account like:
+
+```yml
+algolia:
+  settings:
+    customRanking:
+      - desc(popular)
+      - desc(date)
+      - desc(custom_ranking.heading)
+      - asc(custom_ranking.position)
+```
+
+This will rank popular posts matching your keywords before other posts. You can
+use either boolean or numeric values for the `customRanking`.
+
+[1]: options.html#nodes-to-index
+[2]: https://www.algolia.com/pricing
+[3]: https://www.algolia.com/pricing
+[4]: https://www.algolia.com/pricing
+[5]: https://www.algolia.com/pricing
+[6]: https://www.algolia.com/pricing
+[7]: https://www.algolia.com/pricing
+[8]: https://github.com/algolia/jekyll-algolia/issues
+[9]: https://community.algolia.com/jekyll-algolia/options.html#settings
+[10]: https://www.youtube.com/watch?v=H6crAohtUBw
+[11]:
+  https://github.com/algolia/jekyll-algolia/blob/develop/lib/jekyll/algolia/configurator.rb#L27-L30

docs-src/src/getting-started.md

@@ -10,6 +10,8 @@ layout: content-with-menu.pug
 `jekyll-algolia` is a Jekyll plugin that lets you push all your content in an
 Algolia index.
 
+> While this plugin was created by Algolia, it is not an officially supported API client. It is possible that future major versions of Jekyll break compatibility, or require changes.
+
 ## Requirements
 
 You'll need:

docs-src/src/github-pages.md

@@ -50,6 +50,8 @@ keeping track of the configuration easier.
 # This file should be at the root of your project
 language: ruby
 cache: bundler
+before_install:
+  - gem install bundler
 script:
   - bundle exec jekyll algolia
 branches:
@@ -68,6 +70,16 @@ You might have to edit the `branches.only` value to either `master` or
 `gh-pages`, depending on which branch is configured to be deployed in your
 GitHub Pages configuration.
 
+### Ignoring vendors
+
+Travis bundles all gems in the `vendor` directory on its servers, which Jekyll
+will mistakenly read. This will likely make the process fail. To avoid this,
+add `vendor` to the `exclude` list in your `_config.yml` file.
+
+```yml
+exclude: [vendor]
+```
+
 ## Adding the API Key
 
 The plugin will need your Admin API key to push data. Because you don't want to

docs-src/src/how-it-works.md

@@ -6,8 +6,8 @@ layout: content-with-menu.pug
 # How does this work?
 
 This page will give you a bit more insight about how the internals of the plugin
-are working. This should give you more context to better understand the various
-options you can configure.
+are working. This should give you more context to better understand the options
+you can configure.
 
 ## Extracting data
 
@@ -53,26 +53,34 @@ data. Shared data is the metadata of the page it was extracted from (`title`,
 front-matter). Specific data is the paragraph content and, if applicable, the
 list of parent headings (based on the `<h1>` and `<h6>` of the page).
 
-Using the [distinct setting][1] of the Algolia API, only the best matching
+Using the [distinct setting][1] of the Algolia API, the best matching
 paragraph of each page is returned for a specific query. This greatly improves
 the perceived relevance of the search results as you can highlight specifically
 the part that was matching.
 
+Also note that the response you'll get from the API might be enriched with
+[\_highlightResult][2] and [\_snippetResult][3] keys. Those keys are
+automatically added when performing a search and are not part of the saved
+record.
+
 ## Pushing data
 
 The plugin tries to be smart by using as less operations as possible, to be
-mindful of your Algolia quota. Whenever you run `jekyll algolia`, only records
+mindful of your Algolia quota. Whenever you run `jekyll algolia`, records
 that changed since your last push will be updated.
 
 This is made possible because each record is attributed a unique `objectID`,
 computed as a hash of the actual content of the record. Whenever the content of
-the record changes, its `objectID` will change as well. This allows us to compare
-what is current available in your index and what is about to be pushed, to only
-update what actually changed.
+the record changes, its `objectID` will change as well. This allows us to
+compare what is available in your index and what is about to be
+pushed, to update what actually changed.
 
 Previous outdated records will be deleted, and new updated records will be added
 instead. All those operations are grouped into a batch call, making sure that
 the changes are done atomically: your index will never be in an inconsistent
-state where records are only partially updated.
+state where records are partially updated.
+
 
 [1]: https://www.algolia.com/doc/guides/ranking/distinct/?language=ruby#distinct-to-index-large-records
+[2]: https://www.algolia.com/doc/api-reference/api-methods/search/?language=ruby#method-response-_highlightresult
+[3]: https://www.algolia.com/doc/api-reference/api-methods/search/?language=ruby#method-response-_snippetresult

docs-src/src/layouts/common/footer.pug

@@ -17,7 +17,7 @@ section.footer-new-cta.footer-new.h300.pos-rel
       .button-holder.h200.p-r-large
         .spacer16.hidden-md.hidden-sm
         span.inline.pos-rel
-          a.btn.btn-static-primary.btn-static-shadow-dark(href='https://algolia.com/users/sign_up')
+          a.btn.btn-static-primary.btn-static-shadow-dark(href='https://www.algolia.com/users/sign_up/hacker')
             | Get Started
             i.icon.icon-arrow-right.color-bunting.m-l-small
 

docs-src/src/options.md

@@ -26,16 +26,21 @@ you might want to update the value like this:
 ```yml
 algolia:
   # Also index AsciiDoc and Textile files
-  extensions_to_index: 'html,md,adoc,textile'
+  extensions_to_index:
+    - html
+    - md
+    - adoc
+    - textile
 ```
 
 ## `files_to_exclude`
 
-This option lets you define a blacklist of source files you don't want to index.
+This option lets you define a list of source files you don't want to index.
 
-By default it will exclude all the `index.html` and `index.md` files. Those
-files are usually not containing much text (landing pages) or containing
-redundant text (latest blog articles) so they are not included by default.
+By default it will exclude the `index.html` and `index.md` files found at the
+root. Those files are usually not containing much text (landing pages) or
+containing redundant text (latest blog articles) so they are not included by
+default.
 
 If you want to index those files, you should set the value to an empty array.
 
@@ -46,7 +51,7 @@ algolia:
 ```
 
 If you want to exclude more files, you should add them to the array. Note that
-you can use glob patterns to exclude several files at once.
+you can use glob patterns (`*` and `**`) to exclude several files at once.
 
 ```yml
 algolia:
@@ -57,6 +62,7 @@ algolia:
     - excluded-file.html
     - _posts/2017-01-20-date-to-forget.md
     - subdirectory/*.html
+    - **/*.tmp.html
 ```
 
 _Note that some files (pagination pages, static assets, etc) will **always** be
@@ -100,7 +106,17 @@ algolia:
 ```
 
 Settings defined here will take precedence over any setting you manually defined
-through the [Algolia dashboard][5] UI, though.
+through the [Algolia dashboard][5] UI, though. If you'd like this not to happen
+at all, and only take the dashboard settings in account, pass `false` to the
+settings configuration.
+
+```yml
+algolia:
+  settings: false
+```
+
+We suggest users to at least run with the default settings once, so that the default
+relevance settings are set, which you can override via the dashboard after.
 
 ## `indexing_batch_size`
 
@@ -120,6 +136,24 @@ algolia:
   indexing_batch_size: 500
 ```
 
+## `max_record_size`
+
+**This is an advanced option. It has no effect on Community plans.**
+
+If you're using a paid Algolia plan, you can push records that weight up to 20Kb
+(as opposed to 10Kb with the free Community plan). This option allows you to
+adjust the maximum size of one record (in bytes).
+
+```yml
+algolia:
+  # Recommended setting for paid plans
+  max_record_size: 20000
+```
+
+_Note that if you push a record that is larger than your allowed limit,
+the push will be rejected by the API. This might result in incomplete data being
+uploaded._
+
 
 [1]: ./how-it-works.html
 [2]: http://www.methods.co.nz/asciidoc/

docs-src/src/themes.md

@@ -11,17 +11,23 @@ to add search into your own design.
 
 Some themes already have `jekyll-algolia` embedded by default:
 
-- [Minimal Mistakes][3], by
-  [@mmistakes][4]
+- [Minimal Mistakes][3], by [@mmistakes][4]
+- [GaeBlogx][5], by [@SeraphRoy][6]
+- [Sakura][7] ([live demo][8]), by [@kimfucious][9]
 
 ## Add your own
 
 If you're maintaining a theme using `jekyll-algolia`, feel free to [edit this
-page][5] to add your theme to the list.
+page][10] to add your theme to the list.
 
 
 [1]: ./blog.html
 [2]: https://github.com/jekyll/minima
 [3]: https://mmistakes.github.io/minimal-mistakes/
 [4]: https://github.com/mmistakes
-[5]: https://github.com/algolia/jekyll-algolia/edit/develop/docs-src/src/themes.md
+[5]: https://github.com/SeraphRoy/GaeBlogx
+[6]: https://www.gaeblogx.com/
+[7]: https://github.com/kimfucious/sakura
+[8]: https://sakura.abts.io/
+[9]: https://github.com/kimfucious
+[10]: https://github.com/algolia/jekyll-algolia/edit/develop/docs-src/src/themes.md