CrowGPT: decoding crow calls with AI

CrowGPT: saying it back

The speaking half. A small transformer, 18M parameters, trained from scratch on call bouts cut out of archive recordings, with no human labeling anywhere in the loop. Ask it for an alarm call, a generic call, a flight call or begging, and it produces one.

Hear it

Two generations of the model. Everything under a CrowGPT caption is generated, not a recording of a bird. Next to them, the real thing: a field-recorded American Crow alarm call, labeled by ear, so you can judge the gap, and the progress, yourself.

CrowGPT1: generated, the early “crowbot” stage

CrowGPT2: generated, the current model with a 6-second context window

A real crow: field recording

Two things about it matter more than the audio. It does not copy its training set: what it produces is generated, not retrieved. And it beat a 433M pretrained audio model fine-tuned on the same corpus, which says the corpus and the training setup are doing real work.

Why there is no quality percentage here

The obvious way to score generated calls is to ask BirdNET whether it hears a corvid. That number is circular: BirdNET also selected CrowGPT's training corpus, so the model has been trained indirectly to please the judge scoring it. It scores above real audio, which is not plausible as realism and is how the problem was caught. Every BirdNET-detection figure in this project inherits that bias, so the honest tiebreaker is a listening test, and no number goes here until one is run.

The app: collecting the other half

Building a labeled set by hand does not scale, so the collection is its own system: record corvid audio in the field with GPS, get it analyzed, and say what was heard and what was happening. Heavy model work never sits in the request path.

The CrowGPT app's capture list, showing starred recordings with dates, durations and a coarse location — Record, then keep. The ★ is the recordist saying *this one is real*.

A capture's waveform with colored spans labeled flight call, begging and call — Calls typed and placed on the waveform: flight call, begging, call.

One detected call under review: the analyzer's guess of begging shown above a grid of call types, with begging confirmed, and a row of caller roles below — The analyzer's guess is shown, then confirmed or overruled. Only the human answer counts.

The context picker for a single call, with chips grouped under people, animals, food and what the crows did; two are selected — What was happening, from a closed vocabulary. Tapped per call, not per recording.

A capture named lawnmower plus four crows on ground, with a written description of the scene — The witnessed situation, written at the time. This is the scarce half.

A capture describing crows scattering from a car, with a separate guess field about what it meant — Guesses are kept apart from observations, so being wrong costs nothing.

The app

Android, local-first. Records unprocessed 48 kHz audio and video, shows the waveform, and uploads only when asked, so an interrupted upload is the normal case, and every step is resumable and safe to repeat.

The analysis

Detection and call-typing run in the background, off the recording device. Findings come back attached to the waveform, and nothing waits on a model to finish.

The labeling desk

Where the work happens. Calls get confirmed, corrected, or marked by ear, and each one tagged with what was going on at the time.

A small black shoulder bag with an embroidered crow, a furry windshield poking out of the top — The whole field kit: a shotgun mic, a dead cat, and a bag that fits in one hand. Crows notice equipment.

Three kinds of ground truth, deliberately kept apart

Labels are what the bird said: a span plus a call type from a closed vocabulary. They are the only thing that feeds training. Context is what a human saw while that call was made, and it attaches to the call rather than the recording: tag the whole capture and you credit every situation to every call in it, so a clip where a dog walks through and the crow later sings would report that crows sing at dogs. Observations are free text written afterwards, to explain a confusing prediction and to find which situations recur often enough to earn a tag of their own.

A call can be marked by ear, with no detection behind it

That matters more than it sounds. BirdNET scores a begging fledgling at 0.06, so a system that only lets you confirm detections can only ever collect what the model already finds, and the classes it is worst at could never gain a single training example.

CawScout: where to point the microphone

One backyard produces one family's calls. Finding the next family used to be a random dawn walk, and the random walks kept coming home empty. CawScout replaces them with a prior: a 250 m grid over the whole valley, scored on where crows should be and ranked into a numbered target list on the app's map.

The trees

Crows nest and roost high, so canopy is the loudest signal in the score. A satellite canopy-height model reads every tree in the valley at 1 m resolution; stands of 15 m and up, about 45,000 of them, go into the grid, joined by thousands of individually-mapped street trees, each assigned its height from the same model. The taller the stand, the harder it pulls on the cells around it.

The greenery

Crows forage on open turf, so parks, cemeteries, golf courses and grass come in from OpenStreetMap, nearly eight thousand features, with water and human food (fast food, supermarkets, picnic sites) as their own smaller signals. Each one radiates a few hundred metres into the surrounding cells, because a crow near a park is a crow the park explains.

The score

Every signal becomes a heat layer over the grid and they sum by weight, canopy first, community crow sightings second, on purpose: the satellite measures every block equally, while sightings pile up wherever birders happen to walk. Predators then subtract, corrected for how well-watched the ground is. Great Horned Owls, the top predator of crow nests, cost the most. That penalty alone knocked a raptor-patrolled university cluster out of the top five.

The map

Numbered targets over satellite imagery, alongside every capture's pin and community sightings of crows and each predator species: dated, photographed, toggleable per species. A target explains its own score and offers street view and directions, and the header says whether right now is even a good time to be out.

The result that matters

Two call classes stayed stubbornly weak when trained on the archives' coarse free-text tags. Trained instead on hand-reviewed labels (3,162 segments somebody sat and listened to, one second at a time), the same pipeline, unchanged, lifts them decisively. The bottleneck was never the algorithm.

One-vs-rest AUC by label quality American Crow · recording-grouped cross-validation, so nothing is graded on data it trained on

Chance is 0.50. The two classes not shown, flight call and rattle, were already strong on weak labels; these two were the failures.

So alarm and song were label-limited, not unlearnable. The path to strong heads is clean contextual labels, not a fancier model, which is exactly why the rest of this project is a data-collection system rather than more machine learning.

Crow → human → crow

The end state is a translator that runs in both directions. CawDex, the call-type classifier, already works. The missing ingredient is semantics, what was actually happening when a call was made, and I collect that every day. Pair the two and you get a Crow Translate. Simple calls first: dog spotted, human spotted, food.

The other half is CrowGPT, which generates crow calls instead of reading them. Put the two together and it runs both directions: crow → human → crow. Cross-species conversation, and maybe friendship, with the help of AI.

Crow Translate · listening

What did that call mean?

A recording goes in; the corvids in it are found, named, and each call labeled by type on a timeline. That part works. Turning a type into meaning takes the second ingredient, what was happening at the moment the bird called, which is what I am out collecting.

CrowGPT · speaking

Say it back

A model that generates crow calls rather than reading them, on demand and by type. Early, and getting better. It is the half that makes this a conversation instead of a very elaborate subtitle track.

Simple calls come first, because those are the ones a backyard produces often enough to learn from: a dog walking past, a person arriving, food going down. Nobody is decoding crow grammar here.

Two problems hide in one question

Inside the listening half there is a second split, and keeping it straight is the main design decision in the project. These need different machinery, and only one of them is solved.

Solved What are the call types?: Are the categories acoustically separable at all? That is a plain classification problem on audio, and the answer is yes, given labels good enough to learn from. Everything below this line is engineering.
Open What does each type mean?: Mapping a call to “I see a dog” needs that call paired with witnessed context, written down at the moment it happened. No quantity of audio alone gets there, which is why the rest of this project is a data-collection system. This is the frontier, and it is where I am.

CawDex: reading the calls

The listening half. CawDex is the call-type classifier: audio goes in, and every corvid call in it comes out found, named by species, and labeled by type on a timeline. It is the index the translator will be built on, hence the name.

Raw audio is too high-dimensional to classify directly, so every clip becomes a fixed-length fingerprint first.

Detect

BirdNET finds the corvids in any audio or video file and names the species, with a waveform timeline of where the calls sit.
Embed

Each clip becomes a BirdNET embedding, a fingerprint where acoustically similar calls land near each other. Classical MFCC features and a general audio transformer both sat near chance; a bird-specific model was the unlock.
Pool

Mean, standard deviation and max across the clip's three-second segments, so the feature captures temporal spread, the signature of a repeated, insistent alarm versus a single call.
Classify

A small logistic-regression head per species maps the fingerprint to a call type. At a few hundred examples per species, a regularized linear model beats a deep net.

Where it is today

The state of play, plainly. Every limit below is a piece of work waiting, and most of them are more interesting than the parts that already run.

What it can already tell apart

CawDex sorts every call it finds into one of five classes, the same five the app offers a recordist. Finer field labels fold into them, so a mobbing crow is filed as alarm and a rattle as call. The colours are the ones the app and the dashboard use.

Alarm call

Harsh, fast, repeated, with the whole bird in it.

Something is wrong, and the other crows are being told.

Also filed here: mobbing and scold, the directed, persistent version aimed at a cat, an owl or a hawk on the ground.

Reads it at 0.97 AUC on clean labels.
Flight call

Spaced, carrying caws given on the wing.

Keeps the group in touch while it moves: here I am, this way.

The cleanest class acoustically, and the first one the pipeline learned.

Reads it at 0.91 AUC, even on archive labels.
Call

Unhurried, spaced out, back and forth between birds.

Nothing is happening. The ordinary talk between crows that know each other.

Also filed here: contact call and rattle, the dry, knocking run a crow gives close to another bird.

The largest and vaguest class, which is why the field labels split it.
Begging

A rasping, nasal, repeated caah, flatter than an adult's, and it goes on and on, wings quivering.

A fledgling asking to be fed. Hear it and there is a family group nearby.

Its own class because juvenile calls were otherwise lost inside the vaguest one.

The detector's weak spot: BirdNET scores a begging fledgling at 0.06, so these mostly arrive by hand.
Song

Quiet, odd, rambling, often to itself.

Nobody knows yet. It likely carries the most information of all, and it is out of scope for now because it needs a close microphone.

Easy to miss and worth keeping whenever it turns up.

Reads it at 0.91 AUC on clean labels, 0.56 on archive tags.

Where it still falls short

Human labeling is early. 66 labels across 16 captures, plus 158 context events. Measured as call audio rather than label count, that is about 7.3 minutes, the number that actually limits this project.
Detected calls are guesses. Only human labels are ground truth, and until recently there were none at all.
No call has been translated. The classifier says alarm call; it does not say dog. Nothing here has yet mapped a call to a meaning, because that needs enough calls paired with a witnessed situation, and I am still collecting the first few dozen.
Public uploads aren't open yet. The code is open, but the fleet is one phone on a home network for now.
Scope is loud calls only. Crows also make quiet soft song that likely carries more information, but it needs a close mic and drowns in noise. Broadcast calls are stereotyped, repeated and high-SNR, which is what weak supervision needs.

The murder in question

The crows I feed are five: Thrasher and Ripjaw the parents, Killer and Shredder the fledglings, and Edgar, a yearling who stayed on to help guard the brood. The notes I was already keeping on them turned out, by accident, to be proto-labels of exactly these call types, which is the whole reason a recording and a witnessed situation arrive together here instead of being reconciled weeks later.

A crow perched on a power line under a sycamore, parked cars below — Edgar on the line out front, watching the street.

A fledgling crow walking across a flat white roof, seen through a window — Killer on the office roof, the friendly one of the two.

A scruffy young crow on a concrete ledge beside a window, another crow's tail just visible below — Killer earlier in the summer, still half down.

Two crows perched on the ornamented roofline of a brick building — Two of the family on the roofline, keeping watch.

Three crows crowded together on the arm of a streetlight — Killer, Edgar and Shredder sharing a streetlight, which is most of what they do.

A crow perched on a cable above a pavement, seen from behind — Ripjaw on the wire, waiting for the peanuts to appear.

A crow calling from the parapet of a brick building against a gray sky — Edgar, mid-call, from the top of the brick.

A crow on a power line turned to face the camera directly — Edgar deciding what I am. He usually gets there first.

Killer, Shredder and Edgar working through a handful of peanuts on the office roof.

The family fighting over peanuts on a light post. No sound, no point. Just crows.

Digging in the gravel out front. What they were after, I never worked out.

Killer calls back when I say “bye buddies.” That is not a result. It is why I started.

Open to collaborators

This is a one-person project that does not need to stay one. There is more work here than time, and several parts are self-contained enough to hand to someone who finds them interesting.

Birders and crow people. The scarce resource is not compute, it is somebody who can say what a bird was doing when it called. If you watch corvids closely, you already have the rare skill.
Audio ML. CrowGPT is small and there is obvious headroom in the generation half, and a listening test to design that is not circular.
Android and dashboards. The capture app and the labeling desk are where collection speed is won or lost.
Anyone with corvids nearby, any species, anywhere. This is not an American crow project. Detection already routes to a per-species model, and there are heads for hooded crow, carrion crow, rook and common raven as well. So hooded crows in Germany or eastern Europe, carrion crows in Britain, rooks in a field somewhere: all of it counts, and comparing what the same call type means across populations is a better question than anything one backyard can answer.

The code is open: github.com/benfaerber/crowgpt. Everything on this page is in it, the classifier, the generator, the app, the field server and every numbered experiment, with a quick start that runs on a fresh clone. Issues and pull requests are welcome.

Tell me what you'd want to work on: hello located at cawtalker period org.

What gave me the idea

Two projects convinced me this was worth attempting at all. Neither is mine, and both are further along than anything here.

Carrion crows, biologgers, and 150,000 calls
Vittorio Baglione and Daniela Canestrari at the University of León, with the Earth Species Project, put tiny recorders on wild carrion crows in northern Spain and mapped the repertoire of 24 family groups against what the birds were actually doing. Those crows are cooperative breeders: the young stay home for years and help raise their siblings, which is exactly what Edgar is doing on the roof here. It is the same question as this project, asked with instruments. The preprint.
Project CETI
The Cetacean Translation Initiative is doing this for sperm whales: tags, buoys and drones feeding machine learning, with the stated aim of a blueprint for another species' language. They have proposed a phonetic alphabet for whale codas. If it can be attempted at that scale in the open ocean, it can be attempted in a backyard.