rss logo

Elastic 9 - Delete indices and data streams from the CLI

Elasticsearch logo

Depending on what we monitor, Elasticsearch indices and data streams can take up a lot of disk space and quickly saturate the partition on which they are stored.

As a result, we may end up stuck at the Kibana login screen, unable to access the web interface to free up space.

We will see here how to delete indices and data streams from the command line in order to recover the system and make it work again.

💡 Note: I use an HTTPS connection to interact with Elasticsearch. This is the default setting in version 9. Remove --cacert and use http://localhost:9200 if you use HTTP instead.

The first step is to list the existing indices to identify which ones can be removed.

  • List all indices:
root@host:~# curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic -k -X GET 'https://localhost:9200/_cat/indices/'
Enter host password for user 'elastic':

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
yellow open .ds-filebeat-8.3.1-2022.07.08-000003      cNnv6fl0ThG1CUmhkkzkXA 1 1 8529476 0     5gb     5gb
yellow open cisco-switches-2022.05.12                 0J9FHauHTWeZqQAAN1qXCQ 1 1    7976 0 879.5kb 879.5kb
yellow open .items-default-000001                     v5Gt0zwgQEmQfkMyCKkiGw 1 1       0 0    225b    225b
yellow open .ds-filebeat-8.2.0-2022.07.03-000025      BAHTgl7_Smekbi7Wakn69Q 1 1 9237525 0     5gb     5gb
[…]
yellow open cisco-switches-2022.05.24                 4pEHXcmwTdaMpJMRfQfd0w 1 1   29577 0   2.6mb   2.6mb
yellow open cisco-switches-2022.06.24                 jKrI1eGDQVyLyXhkZhXoQg 1 1   28155 0   2.8mb   2.8mb
yellow open cisco-switches-2022.05.03                 k9fMLjaqQIyQljNj6TC7pg 1 1    8409 0 986.4kb 986.4kb
yellow open cisco-switches-2022.06.18                 UgigNgghQdK4C2H1uqglIw 1 1   23832 0   2.2mb   2.2mb
yellow open cisco-switches-2022.04.26                 NrQ29xXpQ4m5l-gwKcG8oA 1 1    9295 0   1.1mb   1.1mb
yellow open .ds-filebeat-8.2.0-2022.06.27-000019      wdiqmJO6SpmMXcnmTkCSOw 1 1 9003194 0     5gb     5gb
yellow open cisco-switches-2022.06.03                 XzmysXRCT22ydHzIKPYBEg 1 1   29038 0   2.8mb   2.8mb
yellow open .lists-default-000001                     1unfJPZlSs6er-_01tQMKw 1 1       0 0    225b    225b
yellow open cisco-switches-2022.06.16                 uPSAZ6dZTdawYOkIA9hwLw 1 1   21142 0   2.1mb   2.1mb
yellow open cisco-switches-2022.05.04                 z7A1bypTQTWctsPBLYfbcw 1 1    8456 0 971.8kb 971.8kb
  • List large indices expressed in gigabytes:
root@host:~# curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic -k -X GET 'https://localhost:9200/_cat/indices/' | grep gb
Enter host password for user 'elastic':

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12519  100 12519    yellow open .ds-winlogbeat-8.1.1-2022.07.09-000033    xL4BLmbZSv-igD6DC_b9bw 1 1  461545 0 649.4mb 649.4mb
0  yellow open .ds-winlogbeat-8.1.1-2022.07.05-000031    Vfj_XhjBS0Czq-6u669jqw 1 1 4339397 0     5gb     5gb
   yellow open .ds-filebeat-8.2.0-2022.06.26-000017      OBPonsUVQWqPlo_Qmyko1w 1 1 9302724 0     5gb     5gb
0  yellow open .ds-filebeat-8.2.0-2022.06.29-000021      UgvpYxG6T8uFFiLgtamoZA 1 1 9093846 0     5gb     5gb
 2yellow open .ds-filebeat-8.2.0-2022.07.03-000025      BAHTgl7_Smekbi7Wakn69Q 1 1 9237525 0     5gb     5gb
18k  yellow open .ds-filebeat-8.2.0-2022.07.05-000027      c_fbRaAWQLC36ph8xiV3UQ 1 1 6164373 0   3.4gb   3.4gb
 yellow open .ds-winlogbeat-8.1.1-2022.06.27-000027    m0_AG7KXS1Or32nd-C6AKw 1 1 4387161 0     5gb     5gb
 yellow open .ds-filebeat-8.2.0-2022.06.27-000019      wdiqmJO6SpmMXcnmTkCSOw 1 1 9003194 0     5gb     5gb
  yellow open .ds-filebeat-8.3.1-2022.07.07-000002      dLYUzYNWSmiTKi3k6Kpnew 1 1 8621268 0     5gb     5gb
0 -yellow open .ds-filebeat-8.3.1-2022.07.09-000005      QUEITa46TOyvy3WKphoJ7g 1 1 4730233 0   2.6gb   2.6gb
-yellow open .ds-filebeat-8.3.1-2022.07.08-000003      cNnv6fl0ThG1CUmhkkzkXA 1 1 8529476 0     5gb     5gb
:--yellow open .ds-winlogbeat-8.1.1-2022.06.30-000029    jZnFQ1XRT-iY4m5CYA-ugA 1 1 4385964 0     5gb     5gb
:--yellow open .ds-filebeat-8.3.1-2022.07.08-000004      0-EoyIyyRDSA3s1Fw_Vt-A 1 1 9052394 0     5gb     5gb
 --yellow open .ds-filebeat-8.2.0-2022.07.01-000023      cHhmzxF7TVGZcd8FO0g2JA 1 1 9381514 0     5gb     5gb
:yellow open .ds-winlogbeat-8.1.1-2022.06.22-000025    mqoAPRX1TMiqhLJEy05bUQ 1 1 4296126 0     5gb     5gb
--:-- --:--:--  218k
yellow open .ds-filebeat-8.3.1-2022.07.06-000001      EsqvKfuITB6GItuH3Vf6Ng 1 1 8623335 0     5gb     5gb
  • List indices related to a specific data stream or integration, such as winlogbeat:
root@host:~# curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic -k -X GET 'https://localhost:9200/_cat/indices/' | grep winlogbeat
Enter host password for user 'elastic':

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12412  100 12412    0yellow open .ds-winlogbeat-8.1.1-2022.06.30-000029    jZnFQ1XRT-iY4m5CYA-ugA 1 1 4385964 0     5gb     5gb
   yellow open .ds-winlogbeat-8.1.1-2022.07.05-000031    Vfj_XhjBS0Czq-6u669jqw 1 1 4339397 0     5gb     5gb
  0 yellow open .ds-winlogbeat-8.1.1-2022.06.27-000027    m0_AG7KXS1Or32nd-C6AKw 1 1 4387161 0     5gb     5gb
  3yellow open .ds-winlogbeat-8.1.1-2022.07.09-000033    xL4BLmbZSv-igD6DC_b9bw 1 1 1517532 0   1.8gb   1.8gb

Delete indices

Now that we have identified the indices or data streams we want to delete, we can use -X DELETE to remove them and recover disk space.

  • To delete an index, specify its name with the following command:
root@host:~# curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic -k -X DELETE "https://localhost:9200/.ds-filebeat-8.2.0-2022.06.26-000017"

🚨 Note: If the index is currently the write index of a data stream, you may get the following error:

root@host:~# curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic -k -X DELETE "https://localhost:9200/.ds-logs-winlogbeat.log-default-2022.06.27-000027"
Enter host password for user 'elastic':

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"index [.ds-logs-winlogbeat.log-default-2022.06.27-000027] is the write index for data stream [logs-winlogbeat.log-default] and cannot be deleted"}],"type":"illegal_argument_exception","reason":"index [.ds-logs-winlogbeat.log-default-2022.06.27-000027] is the write index for data stream [logs-winlogbeat.log-default] and cannot be deleted"},"status":400}
  • This error means that the index is currently the write index of the data stream. To delete it, we first need to perform a rollover on the data stream:
root@host:~# curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic -k -X POST "https://localhost:9200/logs-winlogbeat.log-default/_rollover"
Enter host password for user 'elastic':

{"acknowledged":true,"shards_acknowledged":false,"old_index":".ds-logs-winlogbeat.log-default-2022.06.27-000027","new_index":".ds-logs-winlogbeat.log-default-2026.07.06-000002","rolled_over":true,"dry_run":false,"lazy":false,"conditions":{}}
  • Once the rollover is complete, the old index is no longer the write index and can be deleted:
root@host:~# curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic -k -X DELETE "https://localhost:9200/.ds-logs-winlogbeat.log-default-2022.06.27-000027"