New York City (NYC) taxi rides are probably the most commonly used benchmark in the area of data analytics. It started with deciding to prepare the collection to analyze 1.1 billion NYC Taxi and Uber Trips. Then continued by testing using the data collection. Todd W. Schneider first in 2015 Mark Litwintschik lots of databases and search engines Now we at have to make it easier to use and made it available as a part of the most transparent and . DB Benchmarks dockerized preparation of the data collection open source database benchmarks suite Data Collection The data collection constitutes 1.7 billion taxi and for-hire vehicle (Uber, Lyft, etc.) trips originating in New York City since 2009. Most of the comes from the NYC Taxi & Limousine Commission. raw data The data collection record includes a lot of different attributes of a taxi ride: Pickup date and time Coordinates of pickup and dropoff Pickup and dropoff location names Fee and tip amount Wind speed, snow depth And many other fields It can be used mostly for testing analytical queries, but it also includes a couple of full-text fields that can be used to test free text capabilities of databases. The whole list of fields and their data types is: "properties": {
         "vendor_id": {"type": "keyword"},
         "pickup_datetime": {"type": "date", "format": "epoch_second"},
         "dropoff_datetime": {"type": "date", "format": "epoch_second"},
         "store_and_fwd_flag": {"type": "keyword"},
         "rate_code_id": {"type": "integer"},
         "pickup_longitude": {"type": "float"},
         "pickup_latitude": {"type": "float"},
         "dropoff_longitude": {"type": "float"},
         "dropoff_latitude": {"type": "float"},
         "passenger_count": {"type": "integer"},
         "trip_distance": {"type": "float"},
         "fare_amount": {"type": "float"},
         "extra": {"type": "float"},
         "mta_tax": {"type": "float"},
         "tip_amount": {"type": "float"},
         "tolls_amount": {"type": "float"},
         "ehail_fee": {"type": "float"},
         "improvement_surcharge": {"type": "float"},
         "total_amount": {"type": "float"},
         "payment_type": {"type": "keyword"},
         "trip_type": {"type": "byte"},
         "pickup": {"type": "keyword"},
         "dropoff": {"type": "keyword"},
         "cab_type": {"type": "keyword"},
         "rain": {"type": "float"},
         "snow_depth": {"type": "float"},
         "snowfall": {"type": "float"},
         "max_temp": {"type": "byte"},
         "min_temp": {"type": "byte"},
         "wind": {"type": "float"},
         "pickup_nyct2010_gid": {"type": "integer"},
         "pickup_ctlabel": {"type": "keyword"},
         "pickup_borocode": {"type": "byte"},
         "pickup_boroname": {"type": "keyword"},
         "pickup_ct2010": {"type": "keyword"},
         "pickup_boroct2010": {"type": "keyword"},
         "pickup_cdeligibil": {"type": "keyword"},
         "pickup_ntacode": {"type": "keyword"},
         "pickup_ntaname": {"type": "text", "fields": {"raw": {"type":"keyword"}}},
         "pickup_puma": {"type": "keyword"},
         "dropoff_nyct2010_gid": {"type": "integer"},
         "dropoff_ctlabel": {"type": "keyword"},
         "dropoff_borocode": {"type": "byte"},
         "dropoff_boroname": {"type": "keyword"},
         "dropoff_ct2010": {"type": "keyword"},
         "dropoff_boroct2010": {"type": "keyword"},
         "dropoff_cdeligibil": {"type": "keyword"},
         "dropoff_ntacode": {"type": "keyword"},
         "dropoff_ntaname": {"type": "text", "fields": {"raw": {"type":"keyword"}}},
         "dropoff_puma": {"type": "keyword"}
       } Databases So far we have made this test available for 3 databases - a powerful OLAP database, Clickhouse - general-purpose “search and analytics engine”, Elasticsearch - “database for search”, Elasticsearch alternative. Manticore Search In this test we make to not give either of them an unfair advantage. Testing at max tuning is no less important, but it's a subject for another benchmark. Here we want to understand what latency a regular non-experienced user can get after just installing a database and running it with its default settings. But to make it fair to compare one with another we still had to change a few settings: as little changes to database default settings as possible Clickhouse: , just and standard docker image. no tuning CREATE TABLE ... ENGINE = MergeTree() ORDER BY id clickhouse-server Elasticsearch: here to make it fair to compare with the other databases we by: had to help Elasticsearch letting it make 32 shards: ( ), otherwise it couldn’t utilize the CPU which has 32 cores on the server, since as in Elasticsearch official guide “Each shard runs the search on a single CPU thread”. "number_of_shards": 32 said since as said on it needs to be done for performance. bootstrap.memory_lock=true https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#_disable_swapping the docker image is standard Manticore Search is also used in a form of : their own docker image + the columnar library they provide as well as with Elasticsearch we 32 shards in a form of 32 plain indexes also use and we use Manticore columnar storage since comparing Manticore’s default row-wise storage vs Clickhouse’s and Elasticsearch’s columnar storages would be not fair on such a large data collection. About caches We've also configured the databases to not use any internal caches. Why this is important: In this benchmark, we conduct an to find out what response time users can expect if they run one of the tested queries at a random moment, not after running the same query many times consequently. accurate latency measurement Any cache is a shortcut to low latency. As written in "cache stores data so that future requests for that data can be served faster". But caches are different, they can be divided into 2 main groups: Wikipedia 👌 those that just cache raw data stored on disk. For example many databases use to map the data stored on disk to memory, access it easily and let the operating system take care about the rest (reading it from disk when there's free memory, removing it from memory when it's needed for something more important etc). This is ok in terms of performance testing, because we let database leverage the benefit of using the OS page cache (or its internal similar cache that just reads data from disk) mmap() each That's exactly what we do in this benchmark. ❗ those that are used to save results of previous calculations. And it's fine in many cases, but in terms of this benchmark letting database enable such a cache is a bad idea, because: it breaks proper measuring: instead of measuring calculation time you start measuring how long it takes to find a value by a key in memory. It's not something we want to do in this test (but it's interesting in general and we'll perhaps do it in the future and publish some article "Benchmark of caches"). even if they save not a full result of a particular query, but results of its sub-calculations it's not good, because it breaks the idea of the test - "what response time users can expect if they run one of the tested queries at a random moment". some databases have such a cache (it's usually called "query cache"), others don't so if we don't disable database internal caches we'll give an unfair advantage to those having that. So we do everything to make sure none of the database does this kind of caching. What exactly we do to achieve that: Clickhouse: , , (not each attempt of the same query). SYSTEM DROP MARK CACHE SYSTEM DROP UNCOMPRESSED CACHE SYSTEM DROP COMPILED EXPRESSION CACHE before testing each new query Elasticsearch: in its configuration "index.queries.cache.enabled": false (not each attempt of the same query). /_cache/clear?request=true&query=true&fielddata=true before testing each new query Manticore Search (in configuration file): qcache_max_bytes = 0 docstore_cache_size = 0 Operating system: we do before each query ( each attempt). I.e. for each new query we: echo 3 > /proc/sys/vm/drop_caches; sync NEW NOT stop database drop OS cache start it back make the very first cold query and measure its time and make tens more attempts (up to 100 or until the coefficient of variation is low enough to consider the test results high quality) Queries The queries are mostly analytical queries that do filtering, sorting and grouping. We’ve also included one full-text query: [
"SELECT count(*) FROM taxi where pickup_ntaname = '0'",
"SELECT pickup_ntaname, count(*) c FROM taxi GROUP BY pickup_ntaname ORDER BY c desc limit 20",
"SELECT cab_type, count(*) c FROM taxi GROUP BY cab_type order by c desc LIMIT 20",
"SELECT passenger_count, avg(total_amount) a FROM taxi GROUP BY passenger_count order by a desc LIMIT 20",
"SELECT count(*) FROM taxi WHERE tip_amount > 1.5",
"SELECT avg(tip_amount) FROM taxi WHERE tip_amount > 1.5 AND tip_amount < 5",
"SELECT rain, avg(trip_distance) a FROM taxi GROUP BY rain order by a desc LIMIT 20",
{
  "manticoresearch": "SELECT * FROM taxi where match('harlem east') LIMIT 20",
  "clickhouse": "SELECT * FROM taxi where match(dropoff_ntaname, '(?i)\\WHarlem\\WEast\\W') or match(pickup_ntaname, '(?i)\\WHarlem\\WEast\\W') LIMIT 20",
  "elasticsearch": "SELECT * FROM taxi where query('harlem east') LIMIT 20"
},
"SELECT avg(total_amount) FROM taxi WHERE trip_distance = 5",
"SELECT avg(total_amount), count(*) FROM taxi WHERE trip_distance > 0 AND trip_distance < 5",
"SELECT count(*) FROM taxi where pickup_ntaname != '0'",
"select passenger_count, count(*) c from taxi group by passenger_count order by c desc limit 20",
"select rain, count(*) c from taxi group by rain order by c desc limit 20",
"SELECT count(*) from taxi where pickup_ntaname='Upper West Side'",
"SELECT * from taxi limit 5",
"SELECT count(*) FROM taxi WHERE tip_amount = 5",
"SELECT avg(total_amount) FROM taxi"
] Results You can find all the results on the by selecting “Test: taxi”. results page Remember that . The other 2 (“Fastest” and “Slowest”) are provided with no guarantee since: the only high-quality metric is “Fast avg” since it guarantees a low and a high query count conducted for each query coefficient of variation - is a single attempt result, in most cases the very first coldest query. Even though we purge OS cache before each cold query it can’t be considered stable. So it can be used for informational purposes only (even though many benchmark authors publish such results without any disclaimer). Slowest - just the very fastest result, it should be in most cases similar to the “Fast avg” metric, but can be more volatile from run to run. Fastest Remember the tests including the results are 100% transparent as well as everything in this project, so: you can use to learn how they were made the test framework and find raw test results in the directory. results Unlike other less transparent and less objective benchmarks we are not making any conclusions, we are just leaving screenshots of the results here: All Three Competitors at Once Clickhouse vs Elasticsearch Manticore Search vs Elasticsearch Manticore Search vs Clickhouse Disclaimer The author of this test and the test framework is a member of core team and the test was initially made to compare Manticore Search with Elasticsearch, but as shown above and can be verified in the and by running the same test yourself Manticore Search wasn’t given any unfair advantage, so the test can be considered unprejudiced. However, if something is missing or wrong (i.e. non-objective) in the test feel free to make a pull request or an issue on . Your take is appreciated! Thank you for spending your time reading this! Manticore Search open source code Github Also published . here

Lyft

Uber

2022 - HackerNoon Contributor of the Year - Database

Read My Stories

Nominated for 2022 - HackerNoon Contributor of the Year - Database

Too Long; Didn't Read

In your car, at home, or at work — Bosch technology shapes many areas of life.

Clickhouse vs Elasticsearch vs Manticore Search Query Times With a 1.7B NYC Taxi Rides Benchmark

Too Long; Didn't Read

Companies Mentioned

Sergey Nikolaev

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

Clickhouse vs Elasticsearch vs Manticore Search Query Times With a 1.7B NYC Taxi Rides Benchmark

Too Long; Didn't Read

Companies Mentioned

Sergey Nikolaev

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES