19 Nov 2024 (age 34) Web

How I made a public archive of my Twitter posts

It's past time to stop giving any oxygen to the ongoing dumpster fire that is Twitter/X. For those of us who joined in the early days, it's sad to walk away from the platform that had been a touchpoint for political organizing and local community-building. But it's not that place anymore, and the sooner we build a home elsewhere, the better (cough, spore.social).

For old times' sake, though, let's start with a trip down memory lane. I joined Twitter in 2009 at the urging of my roommate Isabel. I first approached it as a way to promote my music, and I still wrote posts starting with "is", in the format of OG Facebook status updates: "@samnabi is pumped to play at the circus room on thursday!"

In 2010, I worked as a co-op student in the Ontario government. Canadian politicians like Tony Clement, and journalists like Kady O'Malley and Paul Wells were using Twitter to break news quicker and have direct conversations with regular people. My boss asked me, as the resident millennial, to teach the rest of the team what Twitter was all about. There were already some other government departments using it reliably!

tweet from MBower01, @mnrcentral trying to figure out if there is still a fire-ban in northern ontario. Can you help?

I remember when Twitter had to change their trending topics algorithm away from simply showing the most talked-about topic, because it was just non-stop Justin Bieber. I remember when retweets graduated from a copy-and-paste exercise to a button you could click. I remember when they decided to allow posting photos, instead of just text, on the platform. I remember when they started notifying us when someone favourited a post (so many notifications seemed excessive at the time). Good times.

Today, the times are not so good over at Twitter/X. So I've decided to delete all my posts there, but not before downloading an archive of my posts and hosting them on my own website. Here's how I did it.

Step 1: download your archive

Go to x.com/settings/your_twitter_data to download an archive of all your data. It might take 24 hours or so for the archive to be prepared, and they'll send you an email reminder when it's ready.

This archive is essentially a web app that you can open in your browser, although the data and photos are all stored locally. We'll be modifying the archive to make it safe for public consumption, meaning we will have to strip out a lot of personally-identifying information. If you want to maintain the full version of your archive including all of Twitter/X's tracking and analytics data about you, save another copy of the archive somewhere.

You can view your archive by opening Your archive.html in a browser. It's a read-only version of the Twitter timeline where you can navigate your tweets, retweets, likes, and more.

Step 2: remove ad-related data

In the archive, there's a folder called data/. There's a lot of personally-identifying information in here that I don't want to publish, so let's delete all these files:

Step 3: remove data related to blocked and muted users

It's probably not a good idea to publish a list of people who you've blocked or muted. Let's remove these files from the data/ folder as well:

Step 4: remove private information

There is all kinds of data in the archive that is meant to be private, including identifiers like your email address (including previous email addresses you have used to access the account), IP address, connected apps, direct messages, saved searches, and Grok chats. Let's delete these files from the data/ folder:

Step 5: clean up the manifest file

The data/manifest.js file contains references to a lot of the data that we just deleted. Let's clean up this file by removing references to any of the sensitive data that we dealt with above. Under dataTypes, these are the only items I have kept:

After modifying the dataTypes items, try to visit the archive in your browser to ensure the code is still formatted properly and nothing appears broken.

Step 6: censor personal information in data/account.js

The data/account.js file is necessary for the archive to function properly, but it also contains some information that I don't want to publish. This includes my email address and the method used to create the account. I simply replaced these values with "No data".

You should keep your accountId intact, as it is required for some parts of the archive to display properly.

The full data/account.js file now looks like this:

window.YTD.account.part0 = [
  {
    "account" : {
      "email" : "No data",
      "createdVia" : "No data",
      "username" : "samnabi",
      "accountId" : "19156606",
      "createdAt" : "2009-01-18T20:28:08.000Z",
      "accountDisplayName" : "Sam Nabi – @samnabi@spore.social"
    }
  }
]

Step 7: tweak the archive homepage

This step is optional, I suppose, but if you're publishing your archive for other people to read, it's helpful to give them a bit of an introduction that is different from the message that Twitter/X gives to people who have just downloaded their own archive privately.

  1. I removed the file home-image.png, because I found it unnecessary
  2. In Your archive.html I changed the <title> tag from "Your Twitter Data" to "Sam Nabi's Twitter Archive"
  3. Using CSS, I hid some of the boilerplate text and the menu items for data had already been deleted. I also added a basic introduction by injecting it into a pseudo-element using the content property. I did this through CSS instead of by modifying the content directly, because the archive is already compiled as a JavaScript app and I didn't want to poke around inside minified JS code to change things.

This is the CSS I added to the <style> block near the top of Your archive.html:

      /* Hide inactive menu items */
      a[role="menuitem"][href*="account"],
      a[role="menuitem"][href*="account"],
      a[role="menuitem"][href*="safety"],
      a[role="menuitem"][href*="personalization"],
      a[role="menuitem"][href*="ads"],
      a[role="menuitem"][href*="lists"],
      a[role="menuitem"][href*="messages"] {
        display: none;
      }

      /* Intro title  */
      .css-901oao.r-hkyrab.r-1qd0xha.r-1b6yd1w.r-1vr29t4.r-ad9z0x.r-1yflyrw.r-bcqeeo.r-qvutc0 .css-901oao.css-16my406.r-1qd0xha.r-ad9z0x.r-bcqeeo.r-qvutc0 {
        font-size: 0;
      }
      .css-901oao.r-hkyrab.r-1qd0xha.r-1b6yd1w.r-1vr29t4.r-ad9z0x.r-1yflyrw.r-bcqeeo.r-qvutc0 .css-901oao.css-16my406.r-1qd0xha.r-ad9z0x.r-bcqeeo.r-qvutc0:after {
        font-size: 1rem;
        content: '@samnabi on Twitter (2009-2024)';
      }

      /* Sidebar terms & conditions; intro text */
      .css-1dbjc4n.r-1jgb5lz.r-1ye8kvj.r-1qfoi16.r-tvv088.r-13qz1uu, 
      .css-1dbjc4n.r-1mf7evn.r-tvv088.r-1qewag5 .css-901oao.r-hkyrab.r-1qd0xha.r-n6v787.r-16dba41.r-ad9z0x.r-bcqeeo.r-qvutc0,
      .css-1dbjc4n.r-1j2wfwj .css-1dbjc4n.r-1mi0q7o.r-1j3t67a .css-901oao.r-hkyrab.r-1qd0xha.r-a023e6.r-16dba41.r-ad9z0x.r-bcqeeo.r-qvutc0,
      .css-1dbjc4n.r-1mi0q7o.r-1j3t67a .css-901oao.r-hkyrab.r-1qd0xha.r-a023e6.r-16dba41.r-ad9z0x.r-bcqeeo.r-qvutc0 .css-901oao.css-16my406.r-1qd0xha.r-ad9z0x.r-bcqeeo.r-qvutc0
       {
        display: none;
      }
      .css-1dbjc4n.r-1u4rsef.r-18u37iz.r-1x0uki6.r-15bsvpr.r-13qz1uu .css-1dbjc4n.r-1mf7evn.r-tvv088.r-1qewag5:after {
        content: 'This is a read-only archive of my Tweets. In 2022, I switched to Mastodon as my microblogging community of choice, then removed all my Twitter posts in 2024. These days, you can follow me at @samnabi@spore.social.';
      }

Step 8: redirect image links

For some reason, if you click on an image in the archive, it will open a new tab and take you back to twitter.com. Since we want this archive to exist independently of a decaying platform, I created this script to intercept image clicks and just show you the actual full image instead.

Place this entire <script> block just before the closing </body> tag in Your archive.html.

  <script>
    // On image click, show the local image instead of redirecting to Twitter
    document.addEventListener('click', (event) => {
      if (event.target.classList.contains('Tweet-photo') || event.target.classList.contains('Tweet-photoGalleryPhoto')) {

        // Prevent navigating to the twitter.com URL in image links
        event.preventDefault();

        // Instead, open the image file in a new tab
        window.open(event.target.src, '_blank').focus();
      }
    });
  </script>

Step 9: replace t.co links

All links shared on Twitter/X get shortened using their t.co service. This means that the links in your archive will still depend on Twitter/X's infrastructure to function! Instead, let's find and replace all t.co links with their expanded link.

We can find the expanded link by literally visiting a t.co link and seeing where it sends up. For a small number of links you may be able to do this manually using a find/replace tool in your text editor. For my 19.4k posts, though, I needed an automated solution. Here's a PHP script I used to replace all the t.co links.

Even though I used an automated script here, it took a very very long time to complete. This code is not optimized for speed!

<?php
/**
 * Determines the destination URL of a provided URL that gives a redirect.
 *
 * @param string $url the specified URL
 * @return string $url the destination of the specified URL.
 */
 function getDestinationUrl($url) {
  echo $url;
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, $url);
  curl_setopt($ch, CURLOPT_HEADER, true);
  curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
  curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5); // 5-second timeout
  curl_setopt($ch, CURLOPT_TIMEOUT, 5); // 5-second timeout
  $headers = curl_exec($ch);
  $url = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
  if (strstr($url, '//') !== false) {
    echo " => ".$url."\n";
    return $url;
  } else {
    echo " => null";
    return '';
  }
}

// Define the data files from the twitter archive
$data_files = [
  './data/like.js',
  './data/moment.js',
  './data/tweets.js',
  './data/profile.js'
];

foreach ($data_files as $file) {

  // Load contents of the file into a string
  $file_contents = file_get_contents($file);

  // Find all t.co links and replace them with their actual link
  // This is incredibly slow and not optimized, but I'm only doing this once so whatever :)
  if (preg_match_all('/https?:\/\/t\.co\/[\d\w]+/', $file_contents, $matches)) {
    foreach ($matches[0] as $link) {
      $file_contents = str_replace($link, getDestinationUrl($link), $file_contents);
    }

    // Overwrite the data file with the new replaced contents
    file_put_contents($file, $file_contents);
  }  
}

All done!

Phew, that was a lot of fussing about! Now you should have an archive that you can upload to a public server, without the need to rely on any infrastructure managed by Twitter/X.

You can take a look at mine at samnabi.com/twitter.

After verifying that your archive is looking good, you can go ahead and delete your Tweets using a tool like TweetDelete.

Sam Nabi