Strip your TEXT fields - Exeter Web Feb/2016

strip your `TEXT` fields

Gabi@gabidavila

http://gabriela.io

about• Data Engineer• @Crowdcube

uses• serialised Arrays• JSON• large strings• images*

how does it work?

alter table•creates a temporary table with the new structure

•copies the data from the old table to the new one

•consolidates the new table

example

alter PK: INT -> BIGINT

- 2.147.483.648 TO

2.147.483.647

INT

- 922.337.203.854.775.808 TO

922.337.203.854.775.807

BIGINTIllus

trat

ion

out o

f sca

le

case• > 750 GB• > 380 million lines• 3 TEXT fields• Auto increment: 898.191.090

how long did it take?

2 days

why?

speed

fast slow

storage• 1 file in the disk for each row for each TEXT

field• stored in different location than the table

data itself• each field up to 4 MB

engines

MyISAM• fastest read speed• supports FULLTEXT indexes• non transactional• less data reliability

InnoDB• transactional• better data integrity• until MySQL 5.6 it didn’t support FULLTEXT

indexes in TEXT fields

querying into TEXT fields

• inefficient search with LIKE statement• slow DDL operations (like alter table)• unnecessary increase of the table size

possible alternatives

fastest search

search servers• indexes large bodies• api bindings• decoupled

smart retrieval

content delivery network

• high availability• cheap

TEXT field within the RDBMS

conclusion• there is no silver bullet solution• some implementations may add an

additional layer of complexity to the application

• some implementation works better with decoupled applications

thanks!come say hi :)