A few months ago, I noticed a comment on an RSSHub issue. The author shared their code and approach for using Selenium's built-in GCM to receive the latest updates from Twitter. After reading it, I was deeply impressed and decided to research it myself.
Mozilla Autopush
Mozilla Autopush is a good choice.
Setting Up Subscription
Twitter's push settings are synchronized across all platforms. You only need to perform the action on any logged-in web page/client: follow the account and click the small bell next to it. If it's a client, check All Tweets & Replies/All Posts & Replies or choose based on personal preference.
To reduce interference from other unrelated pushes that might disrupt the script, I recommend going to push settings (Settings -> Notifications -> Preferences -> Push Notifications) and unchecking or disabling all options except for Tweets from people you follow.
I suspect there is a hidden limit on the number of subscribed devices on Twitter. When this limit is reached, older devices might be kicked out. This is supported by the fact that after repeatedly toggling the subscription, my script stopped receiving pushes altogether.
Toggling options may require turning push notifications on and off several times, so I've placed this section at the front.
Getting UAID
When initially connecting to the receiving endpoint, we need to send a JSON immediately, or the connection will be closed.
Please protect this uaid. Anyone with this uaid can receive and manage your subscribed content, and only the last logged-in user with this uaid can receive the pushes.
// first time
↑|{"messageType":"hello","broadcasts":{},"use_webpush":true}
↓|{"messageType":"hello","uaid":"53221452835a4e8000cc9175f0b1af68","status":200,"use_webpush":true,"broadcasts":{}}
// later
↑|{"messageType":"hello","broadcasts":{},"use_webpush":true,"uaid":"53221452835a4e8000cc9175f0b1af68"}
↓|{"messageType":"hello","uaid":"53221452835a4e8000cc9175f0b1af68","status":200,"use_webpush":true,"broadcasts":{}}
Getting Subscription Endpoint
First, we need to generate a channelID, which is a random UUID.
const channelID = crypto.randomUUID()
// -> e.g. a06946f8-8f6f-48f8-8b3f-0d1b2f2dda17
Then, find the VAPID of the publisher. This ID is used to identify the publisher, though it's not particularly useful for ordinary users. The following base64url string is Twitter's VAPID.
BF5oEo0xDUpgylKDTlsd8pZmxQA1leYINiY-rSscWYK_3tWAkz4VMbtf1MLE_Yyd6iII6o-e3Q9TCN5vZMzVMEs
↑|{"channelID":"a06946f8-8f6f-48f8-8b3f-0d1b2f2dda17","messageType":"register","key":"BF5oEo0xDUpgylKDTlsd8pZmxQA1leYINiY-rSscWYK_3tWAkz4VMbtf1MLE_Yyd6iII6o-e3Q9TCN5vZMzVMEs"}
↓|{"messageType":"register","channelID":"a06946f8-8f6f-48f8-8b3f-0d1b2f2dda17","status":200,"pushEndpoint":"https://updates.push.services.mozilla.com/wpush/v2/gAAAAABlhC62rALdnqu3lCUIHoKWomQU3MIGDYCLW8H67XvyCL24kQXhlOUqIt_VHg1Ias0cdqmCC0X9KCD4UH503yPYGjOWhHfwlFw8Xw32xMTPNJb23YqW9Yx6Rxv0MLjG31jlPReYMuRg1N79WX2r5FAQgkKV6a8gfPiBGsEBclgbpx1nTyM"}
This pushEndpoint is the endpoint we need.
Submitting Subscription Information
Next, refer to my previous article to obtain p256dh and auth, then log in to Twitter in the browser. Due to the nature of web scraping, I don't recommend using your main account. Readers can register a secondary account for web scraping.
Open the browser's developer console, paste the following content into the Console, fill in the first three variables (all in base64url), and run. If running in node.js, you also need to add a Cookie. The code is straightforward and shouldn't require much explanation from me.
const p256dh = ''//your ECC public key
const auth = ''//your auth key
const endpoint = ''//push endpoint
// for node.js
//const cookie = {auth_token: '', ct0: ''}
// for browser
const cookie = Object.fromEntries(document.cookie.split('; ').map(cookie => cookie.split('=', 2)))
const push_device_info = {
os_version: 'Windows/Firefox',
udid: 'Windows/Firefox',
env: 3,
locale: 'en',
protocol_version: 1,
token: endpoint,
encryption_key1: p256dh,
encryption_key2: auth,
}
fetch('https://twitter.com/i/api/1.1/notifications/settings/login.json', {
headers: {
//'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0',
'Content-Type': 'application/json',
'x-twitter-auth-type': 'OAuth2Session',
//cookie: 'auth_token=' + cookie.auth_token + ';ct0=' + cookie.ct0 + ';',
'x-csrf-token': cookie.ct0,
'x-twitter-client-language': 'en',
'x-twitter-active-user': 'yes',
authorization: 'Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA',
referrer: 'https://twitter.com/settings/push_notifications'
},
method: 'POST',
body: JSON.stringify({
push_device_info
})
})
Unsubscribing
First, send a request via websocket.
↑|{"messageType":"unregister","channelID":"a06946f8-8f6f-48f8-8b3f-0d1b2f2dda17"}
↓|{"messageType":"unregister","channelID":"a06946f8-8f6f-48f8-8b3f-0d1b2f2dda17","status":200}
Autopush will not verify if the uaid's subscription contains this channelID. As long as a valid UUID is provided, it will return 200.
Then, send a request to Twitter. The first part of the code is from the previous section, so I won't repeat it.
fetch('https://twitter.com/i/api/1.1/notifications/settings/logout.json', {
headers: {
//'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0',
'Content-Type': 'application/json',
'x-twitter-auth-type': 'OAuth2Session',
//cookie: 'auth_token=' + cookie.auth_token + ';ct0=' + cookie.ct0 + ';',
'x-csrf-token': cookie.ct0,
'x-twitter-client-language': 'en',
'x-twitter-active-user': 'yes',
authorization: 'Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA',
referrer: 'https://twitter.com/settings/push_notifications'
},
method: 'POST',
body: JSON.stringify(push_device_info)
})
Receiving Push Notifications
Finally, we've reached the most critical step. Here’s an example copied from earlier.
{
"messageType": "notification",
"channelID": "1ad1e3c9-23b2-42d6-a6e6-9e201628bf8b",
"version": "gAAAAABlgVc3Icf8DQm6I-vltZfKDvi0t1K1jXfEW6t-icQf9Bfp3yROt-QK-oqQ20nL8rMixkkGew2P9ViW-SA4m6TTiJLLOuQ98FV8j-y-Dnb4JTQRcJaN7cLHsV7aYXYH2-RaEpmIddQOJKLv4K3p0sr-1fUi4UJb3UKw8Zu4j3YiOEou4Z9XPdCbaf7ApWN-WugwUzFU",
"data": "5DqqPsQnYgL_SvYJjIeyCEpDxqan7wgaBvJT6pPQ6CArCkgMRGu9tnpc7cJRfWR1arx-GaDlRoKOcTE--AOQ3OAs0Iwcn9eIJnx_5XFiubtEBe5ez1uzbZClw4myn9LGe1NUYCz-Kfx5-qIXO-lLd1eHsBeBcpMyI2wt8fzV1rTtKzgJPo8hLPSckOQ-PDqjZHbbbl8Yu5i2fsXMN_qZcBHYRwmyMOrqlXhN7P4RqvhNQTbZTBzgkA2xIMGneYujY8I8roiq30b6yy-izIp6PE_7y5Bv_E5s_M6pQod6ZqQPjMa2Bjq5M51nyeNG5AYnNwLu8lHfa5SgJif2s7atGar33ibg7keUnfagREUHEVAU388P9toYLN7jIZQmfYGRUa9GNABtWFeRG_qgaTmsneIsr4AU2EEJiJmlWAvV58oWIlRk98kLtJtGRCQmzp29ZgkRJ5RYlwZEVUq9GuA2fLxa48nyj1gNHliSHO39CdrnO9i59OMcJEyXf1IDZOmhV0TVRPCXOOxFF0efLLeRdn16EUHLyb5iyMmlPSJKS4Skd9yM6FAlyUVWp_p1_Ww0nUphGwC2dZPIJyVEcp3t-UzcXmaskhNipmQNvfuF-b0Zz3Fo9yKRl4twFpqGPuEJavaiB_lehHnVBZRrxVh653M-cyKfc63j39J9e8S_aBcuJkr5nebucpC6WY1UCE_ImOidJFuH0mBdy2D677R58qdbLL_KDFyRnaQYMUeaSsQLb1W-cPjY9eQjWgaJGammZ_B8oub4vXt9i6D00lgXk-3bYYhSGEo-KQYzuIHSXAAwEy6LPpxzU4rjJzDKn8IcebSKhEknuU1aMdvIenrbHIiv_3mva6g4CuBGFJp4qTRdZB6VotLrd_vc8HvsNzGQQEugO9U7v9m_GM57RAapsMQU8xi6gyiksb3vShEBxYpVH2bTfN433e-Xb4-S3mG26tD0GQnHss8U3uGBkySd8HTTFAvnIJyr0nXf9z9D4fYG_XIYz21F2lmdMEvSqvs0T9HdEY-xAN986RQ5TGeUwc4Lzaty0Lw9aD00Tq-6XiB8Hi0br1qc0i-R5-sm2bHWZiTJev0MS9-NFUFITpCZ57uFaNTZhAYxxe8t_BtLP392TZXKxNbF9d_xI5A6h1ArusFjCYIF7LTDK6UnNcnhN1-tvVICw7FIhdTEVoev2S_qng7BueWPQBFu84QY0fAM1W7TokfxSeaUs0EQDlT6PxIKrVY9SUpRgooNN_EejClxU4uYUMCHYlKXGKhRFF3OaOFYXNDTPsJ6H9GPjA80Yo8_SEH5ID1VOOf3U7u6n524f93Oiuw0eQ_yK6pUqTT2xKwV6PKMwyGaNd05TCwecw",
"headers": {
"encryption": "salt=FIBbGEB_Pvu9RjiD_eIikg",
"crypto_key": "p256ecdsa=BF5oEo0xDUpgylKDTlsd8pZmxQA1leYINiY-rSscWYK_3tWAkz4VMbtf1MLE_Yyd6iII6o-e3Q9TCN5vZMzVMEs;dh=BAvjLA_jh3hS_0Mb9I0tYzfXo_1onC7_YqEuPh4FOur-2ggugEoRd-_d0zSDh2TWUoh-4JB13eIpw_zEgWyAvXA",
"encoding": "aesgcm"
}
}
First, send an ACK packet.
↑|{"messageType":"ack","updates":[{"channelID":"1ad1e3c9-23b2-42d6-a6e6-9e201628bf8b","version":"gAAAAABlgVc3Icf8DQm6I-vltZfKDvi0t1K1jXfEW6t-icQf9Bfp3yROt-QK-oqQ20nL8rMixkkGew2P9ViW-SA4m6TTiJLLOuQ98FV8j-y-Dnb4JTQRcJaN7cLHsV7aYXYH2-RaEpmIddQOJKLv4K3p0sr-1fUi4UJb3UKw8Zu4j3YiOEou4Z9XPdCbaf7ApWN-WugwUzFU","code":100}]}
For decrypting the received content, please refer to the previous article.
Finally, we get this JSON. The timestamp should be when Autopush received the push, and the initial language setting (zh-tw) seems to have little effect on the content.
{
"registration_ids" : ["https://updates.push.services.mozilla.com/wpush/v2/gAAAAABlgEsJR6WeYoTxZKdbf1GNO1IuKSRyorx8VtnpJqLeLmHOLrKYrI6ToQha8_VTbbCPem-Af9YDFm7TOSeVzPe2aqaqwV0WR34M5CHq4vVukAbC2aM8qriscH8bbGD2vj6Q_glpLyn1lumQQGKuNCgXcKO2-ZPSkcsjR4FRavfL7jqJA0s"],
"title": "BBC News (UK)",
"body": "Llanberis mountain rescuers face burnout after busiest year bbc.in/4756iUr",
"icon": "https://pbs.twimg.com/profile_images/1529107486271225859/03qcVNIk_reasonably_small.jpg",
"timestamp": "1702975287450",
"tag": "tweet-1737030363371716721",
"data": {
"lang": "zh-tw",
"bundle_text": "{num_total, number} 個新{num_total, plural, other {互動}}",
"type": "tweet",
"uri": "/BBCNews/status/1737030363371716721",
"impression_id": "<SUBSCRIBER_TWITTER_UID>-<UNKNOWN_NUMBER>",
"title": "BBC News (UK)",
"body": "Llanberis mountain rescuers face burnout after busiest year bbc.in/4756iUr",
"tag": "tweet-1737030363371716721",
"scribe_target": "tweet"
}
}
Others
- The Autopush server sends a ping frame every 5 minutes, requiring a pong response. If no ping frame is received in time, the connection is broken, and a reconnection is needed. My example sets the threshold at 6 minutes. The browser handles ping frames automatically.
- Sometimes Autopush broadcasts
"remote-settings/monitor_changes", a timestamp string that can be added to the hello request. Its specific use is unclear.// broadcast ↓|{"messageType":"broadcast","broadcasts":{"remote-settings/monitor_changes":"\"1703239035420\""}} // next hello ↑|{"messageType":"hello","broadcasts":{"remote-settings/monitor_changes":"\"1703044634233\""},"use_webpush":true,"uaid":"53221452835a4e8000cc9175f0b1af68"} ↓|{"messageType":"hello","uaid":"53221452835a4e8000cc9175f0b1af68","status":200,"use_webpush":true,"broadcasts":{"remote-settings/monitor_changes":"\"1703239035420\""}} - This approach suits users not sensitive to timeline completeness and message immediacy due to many limitations:
- Pushes are delayed, especially when Twitter pushes the message to the push service. For ordinary accounts, it’s about 100ms, but for accounts with more followers, it ranges from 1 to 100s or more.
- *Self-replies are not pushed, but exceptions have been observed, so further observation is needed.
- *Retweets do not trigger pushes, likely because pushes are based on tags, and retweets use the original tweet’s tag.
- @BBCNews tweets frequently, making it a good observation target.
- Replies do not include the
@prefix and cannot be identified as replies from the content. - Contains no media (images/videos).
- Some users complain about not receiving reply notifications, so shadowbanned accounts may not receive pushes.
Considering client-side pushes as clients push tweets with images. Once the tweet_id is obtained, there are many ways to get the full tweet. Since this method isn’t highly real-time, a few extra seconds don’t matter.- On 2023-12-21, Twitter's timeline API was down for a couple of hours, but the push service continued to operate normally.
- There may be a limit on the number of follows and subscribes, but I don't know the limit and have no interest in finding it.
- It’s unclear how long a subscription lasts.
- Long periods of inactivity (detected via ACK) might cause Twitter to stop pushing. The duration of this inactivity is unknown.
- From Twitter’s perspective, all requests come from a normal browser or client, so it is theoretically very safe.
- Since frequent scraping is unnecessary, it reduces the load on Twitter’s servers.
- This method should theoretically work for any site using Web Push to push the latest content.
- Autopush connections do not restrict cross-domain requests, so imagine the possibilities. Please do not abuse Autopush to avoid restrictions for everyone.
- Example of Twitter Monitor.
- For online tools, see here.
*GCM
Initially, I thought about receiving GCM messages directly without Selenium. After some digging, I found an article explaining the method. A year ago, a blogger shared how they used push-receiver to receive Google Voice notifications. I tried copying and pasting the code, but naturally, it didn’t work.
There might be something I’m missing, and I plan to continue researching this in the future.
*Receiving Push Notifications Directly
Since GCM is not an option, can I set up my push server? I tried setting up a backend for receiving notifications, but after several days, I only received requests from bots scanning for internet vulnerabilities. Perhaps Twitter has a filter list for push service providers.