Filtering

Filters select points by their metadata, in a MongoDB-style JSON syntax. The same filter works in a search, a recommendation, a count, a delete and a metadata update, over REST, gRPC and every client. During a search the index applies the filter as it goes, so you get up to k matching points.

A collection to filter

The examples on this page use four films and a helper that lists which ones a filter matches:

from quiver import Client, Flat
 
client = Client("http://localhost:6900")
films = client.create_collection("films", dimension=2, metric="euclidean", index=Flat())
films.upsert([
    ("alien", [0.0, 1.0], {"title": "Alien", "genre": "sci-fi", "year": 1979, "rating": 8.5,
                           "tags": ["classic", "horror"], "director": {"country": "UK"}}),
    ("arrival", [1.0, 0.0], {"title": "Arrival", "genre": "sci-fi", "year": 2016, "rating": 7.9,
                             "tags": ["drama"], "director": {"country": "CA"}}),
    ("amelie", [1.0, 1.0], {"title": "Le Fabuleux Destin d’Amélie Poulain", "genre": "romance", "year": 2001, "award": "César",
                            "tags": ["classic"], "director": {"country": "FR"}}),
    ("heat", [2.0, 2.0], {"title": "Heat", "genre": "crime", "year": 1995, "rating": 8.3}),
])
 
def matching(filter):
    """Ids of every film that matches `filter`, alphabetically."""
    return sorted(hit.id for hit in films.search([0.0, 0.0], k=10, filter=filter))

Over REST, the same filter goes in the request body's filter field:

curl -X POST localhost:6900/collections/films/points/count -H 'content-type: application/json' \
  -d '{"filter": {"genre": "sci-fi", "year": {"$gt": 2000}}}'

{"count":1}

Equality and comparison

A bare value means equality. An object whose keys all start with $ is a set of operators, and all of them must hold. Several fields in one filter must all match.

print(matching({"genre": "sci-fi"}))
print(matching({"year": {"$gte": 1990, "$lt": 2010}}))
print(matching({"genre": {"$ne": "sci-fi"}, "year": {"$gt": 2000}}))
print(matching({"rating": {"$lt": 8}}))

['alien', 'arrival']
['amelie', 'heat']
['amelie']
['arrival']

print(matching({"genre": {"$in": ["crime", "romance"]}}))
print(matching({"genre": {"$nin": ["sci-fi"]}}))
print(matching({"award": {"$exists": True}}))
print(matching({"rating": {"$exists": False}}))

['amelie', 'heat']
['amelie', 'heat']
['amelie']
['amelie']

Logic

$and and $or take a non-empty array of filters; $not takes one filter.

print(matching({"$or": [{"genre": "crime"}, {"rating": {"$gt": 8.4}}]}))
print(matching({"$not": {"genre": "sci-fi"}}))
print(matching({"$and": [{"tags": "classic"}, {"year": {"$lt": 2000}}]}))

['alien', 'heat']
['amelie', 'heat']
['alien']

Nested fields and arrays

A dotted name walks into nested objects: director.country is the country key of the director object.

When the field holds an array, a condition matches if the whole array satisfies it or any element does. So {"tags": "classic"} matches every film tagged classic, while an array value asks for that exact array:

print(matching({"director.country": "UK"}))
print(matching({"tags": "classic"}))
print(matching({"tags": ["classic"]}))

['alien']
['alien', 'amelie']
['amelie']

$ne and $nin are the negations: on an array they match only when no element (and not the array itself) equals the value.

Three operators look at the array as a whole:

print(matching({"tags": {"$all": ["classic", "horror"]}}))
print(matching({"tags": {"$size": 1}}))
print(matching({"tags": {"$elemMatch": {"$gte": "d", "$lt": "h"}}}))

['alien']
['amelie', 'arrival']
['arrival']

$elemMatch keeps conditions together on one element. For an array of objects, {"cast": {"$elemMatch": {"name": "Weaver", "role": "lead"}}} needs one cast member who is both, where {"cast.name": ...} style conditions cannot express that. Its filter may use $and, $or and $not too.

Text and patterns

As elsewhere, on an array of strings any element may match.

print(matching({"title": {"$text": "amélie destin"}}))
print(matching({"title": {"$text": "AMELIE"}}))
print(matching({"title": {"$prefix": "A"}}))
print(matching({"genre": {"$regex": "^(sci|cri)"}}))
print(matching({"title": {"$regex": "(?i)^a"}}))

['amelie']
[]
['alien', 'arrival']
['alien', 'arrival', 'heat']
['alien', 'arrival']

$text splits both the query and the field into words at every character that is not a letter or digit, and lowercases them. Word order does not matter. There is no stemming or accent folding, so AMELIE does not match Amélie. It is a filter, not a ranking: for keyword relevance, use sparse vectors.

$regex uses Rust's regex syntax. Anchor it with ^ and $ to match the whole string, and add (?i) for case-insensitive matching. Matching always takes linear time, so a pattern cannot stall the server.

Geo

A geo point is an object {"lat": ..., "lon": ...} with latitude in -90..=90 and longitude in -180..=180. A field holding an array of geo points matches if any of them does. Other values never match.

curl -X POST localhost:6900/collections -H 'content-type: application/json' \
  -d '{"name": "stations", "dimension": 1, "metric": "euclidean"}' > /dev/null
curl -X POST localhost:6900/collections/stations/points -H 'content-type: application/json' -d '{"points": [
  {"id": "brussels", "vector": [1], "metadata": {"loc": {"lat": 50.8456, "lon": 4.3571}}},
  {"id": "antwerp",  "vector": [1], "metadata": {"loc": {"lat": 51.2172, "lon": 4.4211}}},
  {"id": "ghent",    "vector": [1], "metadata": {"loc": {"lat": 51.0357, "lon": 3.7106}}},
  {"id": "liege",    "vector": [1], "metadata": {"loc": {"lat": 50.6245, "lon": 5.5667}}}
]}' > /dev/null
curl -X POST localhost:6900/collections/stations/search -H 'content-type: application/json' -d '{
  "vector": [1], "with_metadata": false,
  "filter": {"loc": {"$geoRadius": {"center": {"lat": 50.8456, "lon": 4.3571}, "meters": 45000}}}
}'

{"results":[{"distance":0.0,"id":"brussels"},{"distance":0.0,"id":"antwerp"}]}

stations = client.collection("stations")
box = {"top_left": {"lat": 51.3, "lon": 3.5}, "bottom_right": {"lat": 50.9, "lon": 4.5}}
print(sorted(h.id for h in stations.search([1.0], filter={"loc": {"$geoBox": box}})))

['antwerp', 'ghent']

top_left must not be south of bottom_right. A box whose top_left longitude is greater than its bottom_right one crosses the antimeridian, so 170 to -170 covers the 20 degrees around it.

Types and numbers

• Range operators compare numbers with numbers and strings with strings (byte-wise, so "sci-fi" > "r"). A number never compares with a string, so the condition fails.
• Integers compare exactly, even beyond 2⁵³; floating point is used only when either side is a float. 2016 and 2016.0 are equal.
• Nested arrays and objects compare by value with the same number rules.

print(matching({"genre": {"$gt": "r"}}))
print(matching({"year": {"$gt": "2000"}}))
print(matching({"year": 2016.0}))

['alien', 'amelie', 'arrival']
[]
['arrival']

Exact integers matter for ids stored in metadata. 9007199254740992 and 9007199254740993 are the same double, but not the same filter value:

ids = client.create_collection("ids", dimension=1, metric="euclidean")
ids.upsert([("p", [1.0], {"user": 9007199254740993})])
print(ids.count({"user": 9007199254740993}), ids.count({"user": 9007199254740992}))

1 0

Missing fields, empty and invalid filters

• A missing field fails every condition except $ne, $nin and {"$exists": false}.
• {} matches every point; a JSON null filter means no filter. A delete refuses {} (see data operations).
• A filter must be a JSON object. Unknown operators, a wrongly typed argument, an empty $and/$or/$all/$elemMatch, a $text without words, an invalid $regex, an out-of-range geo point or an empty field name fail with 400 invalid_filter:

curl -X POST localhost:6900/collections/films/points/count -H 'content-type: application/json' \
  -d '{"filter": {"year": {"$between": [1990, 2000]}}}'

{"error":{"code":"invalid_filter","message":"invalid filter: unknown operator `$between`"}}

print(matching({}))

['alien', 'amelie', 'arrival', 'heat']

Field indexes

Without an index, a filter is checked against every point. That costs little for a search, which stops once it has k hits, but count, deletes and metadata updates by filter read the whole collection. Very selective filters also hurt HNSW searches: the graph walk may give up before it reaches enough matches, and return fewer than k.

Index the fields you filter on most:

curl -X POST localhost:6900/collections/films/indexes -H 'content-type: application/json' \
  -d '{"field": "genre"}'
curl -X POST localhost:6900/collections -H 'content-type: application/json' \
  -d '{"name": "docs", "dimension": 384, "field_indexes": ["user_id", "stats.views"]}'

films.create_field_index("genre")
client.create_collection("docs", 384, field_indexes=["user_id", "stats.views"])

A field is a dot-separated path, as in filters; arrays are indexed by element. Up to 32 fields per collection. DELETE /collections/{name}/indexes/{field} drops one, and field_indexes in the collection's info lists them.

An index answers equality, $in, $all, $gt/$gte/$lt/$lte on numbers and strings, $prefix, $text (the index keeps the words of each string) and {"$exists": true}. For $regex, $size, $elemMatch and the geo operators it narrows the points to those that have the field, and checks each of them. In an $and (or an object with several fields), any indexed condition narrows the points looked at; an $or needs every branch indexed. $ne, $nin, $not and {"$exists": false} are checked point by point, as are conditions on fields without an index.

Indexes never change results: the full filter is still checked on every point an index returns. When the index narrows a search to 20,000 points or fewer, Quiver scores those points directly instead of searching the vector index, so selective filtered searches become exact.

The list of indexed fields is saved with the collection and carried by snapshots. The indexes themselves are rebuilt from the points when the server starts, which takes a little time on large collections.