RTMP Live Streaming Protocol Deep Dive — From Basics to Advanced Optimization

When working with RTMP live streaming, understanding the protocol and optimization techniques is essential for stream quality and user experience. This article explores RTMP in depth, covers issues you may hit in real development, and shares practical optimization tips.

We cover RTMP basics and flow — connection setup, data transfer, and session teardown — plus signaling, chunking, and flow control. The article ends with Enhanced RTMP, which extends RTMP to H.265, AV1, and other video codecs.

1. Introduction

RTMP stands for Real Time Message Protocol. Adobe defined it as an application-layer protocol for multiplexing and packetizing multimedia streams. As live streaming and VR grew, RTMP became widely used again among streaming developers.

RTMP Protocol Basics

1. An application-layer protocol built on TCP

2. Default port 1935

RTMP URL Format

rtmp://ip:[port]/appName/streamName
Example: rtmp://192.168.178.218:1935/live/stream

See also: https://blog.csdn.net/ai2000ai/article/details/72771461

2. RTMP Handshake

The RTMP handshake can be simple or complex. Most Adobe RTMP products today use the complex handshake, which is not covered here.

Handshake Packet Format

 0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
|     version   |
+-+-+-+-+-+-+-+-+
 C0 and S0 bits

C0 and S0: 1 byte containing the RTMP version. The current RTMP protocol version is 3.

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           time (4 bytes)                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           zero (4 bytes)                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           random bytes                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           random bytes                        |
|                               (cont)                          |
|                               ....                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                        C1 and S1 bits

C1 and S1: 4-byte timestamp, 4 bytes of 0, and 1528 bytes of random data

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                          time (4 bytes)                       |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                          time2 (4 bytes)                      |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                          random echo                          |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                          random echo                          |
 |                             (cont)                            |
 |                              ....                             |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                            C2 and S2 bits

C2 and S2: 4-byte timestamp, 4 bytes with the peer's timestamp, and 1528 bytes of random data

RTMP Handshake Flow

+-------------+                            +-------------+
|   Client    |      TCP/IP Network        |     Server  |
+-------------+             |              +-------------+
       |                    |                     |
Uninitialized               |                Uninitialized
       |        C0          |                     |
       |------------------->|           C0        |
       |                    |-------------------->|
       |        C1          |                     |
       |------------------->|           S0        |
       |                    |<--------------------|
       |                    |           S1        |
  Version sent              |<--------------------|
       |        S0          |                     |
       |<-------------------|                     |
       |        S1          |                     |
       |<-------------------|               Version sent
       |                    |           C1        |
       |                    |-------------------->|
       |        C2          |                     |
       |------------------->|           S2        |
       |                    |<--------------------|
    Ack sent                |                   Ack Sent
       |        S2          |                     |
       |<-------------------|                     |
       |                    |           C2        |
       |                    |-------------------->|
Handshake Done              |               Handshake Done
      |                     |                     |
          Pictorial Representation of Handshake

1. The handshake starts when the client sends C0 and C1. After receiving C0 or C1, the server sends S0 and S1.
2. When the client has both S0 and S1, it sends C2. When the server has both C0 and C1, it sends S2.
3. When the client receives S2 and the server receives C2, the handshake is complete.

Notes:

In practice, the client usually sends C0 and C1 together. After receiving C1, the server often sends S0, S1, and S2 at once. S2 echoes the received C1. The client then receives S1 and echoes it back to the server to finish the simple handshake. The RTMP spec also requires verifying that C1 matches S2; many implementations skip this check. {: .prompt-tip }

The handshake accomplishes two things:

1. Verify the RTMP protocol version on client and server
2. Exchange random data to probe network conditions

3. RTMP Messages

RTMP Message Format:

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 | Message Type |               Payload length                   |
 |   (1 byte)   |                   (3 bytes)                    |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                          Timestamp                            |
 |                          (4 bytes)                            |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                          Stream ID            |
 |                          (3 bytes)            |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                         Message Header

1. 1-byte message type

2. 3-byte payload length

3. 4-byte timestamp

4. 3-byte Stream ID to distinguish message streams

Notes:

In real RTMP traffic, messages are not sent in this full form. They are split into chunks, described below. {: .prompt-tip }

RTMP Message Chunking

For a TCP-based protocol, RTMP can feel verbose, but there is no widely adopted simpler replacement. Chunking is one of the trickier parts because it interacts with AMF (also from Adobe).

RTMP Chunk Format

 +--------------+----------------+--------------------+--------------+
 | Basic Header | Message Header | Extended Timestamp |  Chunk Data  |
 +--------------+----------------+--------------------+--------------+
 |                                                    |
 |<------------------- Chunk Header ----------------->|
                            Chunk Format

An RTMP chunk consists of:

1. Basic Header
2. Message Header
3. Extended Timestamp
4. Chunk Data

There are three Chunk Basic Header formats:

Format 1:

   0 1 2 3 4 5 6 7
  +-+-+-+-+-+-+-+-+
  |fmt|   cs id   |
  +-+-+-+-+-+-+-+-+
 Chunk basic header 1

Format 2:

  0                      1
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |fmt|      0    |  cs id - 64   |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      Chunk basic header 2

Format 3:

  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |fmt|         1 |          cs id - 64           |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
             Chunk basic header 3

Notes:

fmt: selects the Message Header type inside the Chunk Header (see below)
cs id: chunk stream id. Chunks from the same RTMP message share the same cs id. The number of bytes for cs id depends on the chunk basic header format {: .prompt-tip }

Message Header Formats

The Message Header type is set by fmt in the chunk basic header. There are four types:

Format 0:

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                          timestamp            | message length|
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |      message length (cont)    |message type id| msg stream id |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |          message stream id (cont)             |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                Chunk Message Header - Type 0

The Message Header is 11 bytes. The first chunk of a chunk stream must use this format.

• timestamp: 3 bytes, max value 16777215 = 0xFFFFFF = 2^24 - 1. When exceeded, all three bytes are set to 1 and the real timestamp goes in Extended Timestamp.

• message length: 3 bytes — the total payload length of the message (e.g. audio or video frame data), not the length of this chunk's data alone.

• message type id: 1 byte — payload type, e.g. 8 for audio, 9 for video.

• msg stream id: 4 bytes — stream ID for this chunk, little-endian (independent of Basic Header CSID)

Format 1:

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                          timestamp            | message length|
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |      message length (cont)    |message type id|  
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
                Chunk Message Header - Type 1 

The Message Header is 7 bytes — it omits the 4-byte msg stream id when this chunk belongs to the same stream as the previous chunk.

• timestamp delta: 3 bytes — difference from the previous chunk's timestamp. Like timestamp, when it overflows, all three bytes are set to 1 and the value is stored in Extended Timestamp.

Format 2:

  0                   1                   2     
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
 |                   timestamp                   |  
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
          Chunk Message Header - Type 2 

The Message Header is 3 bytes — it also omits message length and message type when they match the previous chunk. The three bytes represent timestamp delta, same rules as Format 1.

Format 3:

0 bytes — the Message Header is identical to the previous chunk and is not sent again.

Extended Timestamp

A chunk may carry timestamp or timestamp delta. Extended Timestamp is used only when either value exceeds 0xFFFFFF = 16777215. Otherwise this field is not sent (thanks to @hijiang for the correction).

Extended Timestamp is 4 bytes, max 0xFFFFFFFF = 4294967295. When used, timestamp or timestamp delta must be all 1s so the peer reads the extended field. It stores the full value, not a delta from the truncated field.

Chunk Data: application payload, length in (0, chunkSize]. Default chunk size is 128 bytes.

RTMP Chunking Notes

• Chunk Size

RTMP splits messages by chunk size, which refers to chunk payload only — not Basic Header or Message Header. Client and server each track two values: their own chunk size and the peer's. Both default to 128 bytes. Send a set chunk size message to change the peer's chunk size.

• Chunk Type

RTMP chunks have four types, selected by the top two bits (fmt) of the chunk basic header. A common approach is to send the first chunk of a message as type 0 and subsequent chunks as type 3. Many implementations do this.

If a second message shares the same message stream ID as the first and is longer than chunk size, how do you split it? Documentation is sparse. Implementations such as SRS and FFmpeg use one type 1 chunk for the second message, then type 3 for the remainder — FFmpeg follows this pattern.

RTMP Control Messages

Publish flow: <img src="https://img.pixpark.net/image-20231106131849512-20240114114144762.png" width="50%" height="auto" style="display: block; margin: 0 auto;"> The RTMP spec illustrates publishing as above. Publishing notes:

Connect Message

RTMP command message format:

 +----------------+---------+---------------------------------------+
 |  Field Name    |   Type  |               Description             |
 +--------------- +---------+---------------------------------------+
 |   Command Name | String  | Name of the command. Set to "connect".|
 +----------------+---------+---------------------------------------+
 | Transaction ID | Number  |            Always set to 1.           |
 +----------------+---------+---------------------------------------+
 | Command Object | Object  |  Command information object which has |
 |                |         |           the name-value pairs.       |
 +----------------+---------+---------------------------------------+
 | Optional User  | Object  |       Any optional information        |
 |   Arguments    |         |                                       |
 +----------------+---------+---------------------------------------+

After the handshake, the client sends a connect command. The spec does not mandate exact fields; in practice you send the appName from the RTMP URL plus codec information in AMF. Wireshark capture of typical connect parameters:

The spec does not document every field in detail; see librtmp, srs-librtmp, and packet captures.

The server replies with a _result command. Payload is usually under 128 bytes, though newer nginx-rtmp may return longer responses.

transactionID identifies command messages. The server's _result matches the command via transactionID. After connect, other commands must use different transaction IDs.

After connect, clients often send set chunk size; it is optional.

Window Acknowledgement Size sets the receive window, commonly 2500000 bytes — the peer should send an ACK after receiving that much data. When publishing, the client receives little data from the server, so this message is often omitted. Server ACKs are also often ignored.

Wait for the server's connect response — read all returned chunks, reassemble the RTMP message, and proceed if there is no error.

Create Stream Message

After connecting, create an RTMP stream: send releaseStream, then FCPublish, then createStream.

Parsing the server response to createStream yields a stream ID.

Publish Stream

The last publish step is Publish Stream: send a publish command with the stream name (streamName from the RTMP URL). Use the stream ID from createStream. After publish, many clients send A/V RTMP packets without waiting for a response. Some libraries also send setMetaData (optional), with resolution and other metadata.

Full RTMP publish capture

Once setup is done, send audio and video. Payloads use FLV tag format — see the FLV spec. Incorrect packaging prevents players from decoding A/V correctly.

5. RTMP Timestamps

RTMP timestamps are in milliseconds. They start at zero before A/V is sent, then must increase monotonically according to frame intervals. Inaccurate timestamps can cause A/V sync issues. In srs-librtmp, when pushing a file, H.264 DTS is used as the timestamp. For live capture, a base system time is taken once; each frame's timestamp is the delta from that base.

6. Chunk Stream ID

Chunk Stream ID identifies which message a chunk belongs to. 0 and 1 are reserved. Each new RTMP message type should use a different chunk stream ID — e.g. after a command message, video must use a different ID. The first chunk of each message type must be Type_0.

7. Enhanced Rtmp

Enhanced RTMP extends the protocol. Since March 2023 it supports HEVC (H.265), VP9, AV1, and more. See the Enhanced Rtmp project.

OBS (29.1+) can push H.265 over RTMP — download

FFmpeg plans Enhanced RTMP support; see Liu Qi's talk: FFmpeg live streaming and low-latency progress

As of testing, common players (VLC, IINA, ffplay) could not play H.265 RTMP directly. You can build ffplay for H.265 RTMP — see ffmpeg_rtmp_h265 and Building ffplay to play RTMP H.265

For H.265 RTMP servers, SRS is recommended — see SRS HEVC

8. Summary

RTMP is verbose and nontrivial to implement, but the session flow is relatively straightforward. Many details are underspecified in production; keeping the points above in mind usually keeps A/V working. Corrections and discussion are welcome.

9. Reference Projects

rtmp-publish-kit: pure Java RTMP implementation — Android capture, encode, and RTMP publish
Project: https://github.com/pixpark/rtmp-publish-kit

10. References

1. RTMP Protocol Explained
2. Streaming Live with libRTMP: Pushing AAC and H.264