Magento Indexes

Magento Indexers

Ivan ChepurnyiMagento Trainer / Lead Developer

Ivan Chepurnyi Magento Developers Meetup

Agenda

• Overview of Indexes Functionality• Creation of own indexes


Let Imagine…

… that Magento doesn’t have indexes:

• The prices in product list are calculated on the fly depending on catalog rules, tier prices for customer groups

• Stock availability for configurable and bundle products can be calculated only after loading the product collection

• Layered navigation data is build in real-time for product attributes information

• Anchor categories recursively collects subcategories for filtering product list


It’s all about performance…

The main goal is minimizing amount of operations to display

products to a customer


Definitions

• Indexed DataAggregated data for entity representation on the frontend lists.

• Indexer

Generates index data on event or manual by process.• Index Event

The moment when entity or related to it information is changed and that affects its index data.

• Index Process

Wrapper for indexer and contains information about its mode and status

• Main Controller

Forwards events to Index Process


Index Workflow

EventEventEvents

Process

Main Controller

Indexer

ManualInvoke

Indexed Data


Event Types

• Save

When indexed entity or related to it information was changed

• Delete

When indexed entity or related to it one was deleted

• Mass UpdateWhen batch of entities was updated. (Update Attributes on Product Grid)


Observed Entities

Indexed Entities– Product– Product Inventory– Category– Tag

Entities Scope– Customer Group– Website– Store Group– Store View


Index Process

Available Statuses

• Pending

Indicates that indexer is up to date

• Running

Index currently in process of full rebuilding index data.

• Require ReindexStatus for notifying admin user, that index is not up to date and should be rebuild.


Index Process

Indexing Modes• Real-time

• Manual

Event

EventRequire Reindex

Update Index Data


Indexer Flow

Match Event

Register Event Data

Reindex Data

Main C

ontroller

Index Process


Mage_Index Module

Main Controller

Process

Indexer Base

Mage_Index_Model_Indexer

Mage_Index_Model_Process

Mage_Index_Model_Indexer_Abstract


Catalog Module

Index Module

Indexers Modularity

Mage_Index_Model_Indexer_Abstract

Mage_Catalog_Model_Product_Indexer_Eav

Mage_Catalog_Model_Product_Indexer_Flat

Mage_Catalog_Model_Product_Indexer_Price

Inventory Module

Mage_CatalogIndex_Model_Indexer_Stock

…


ModelMage_Index_Model_Indexer_Abstract

Indexer Structure

Resource ModelMage_Index_Model_Mysql4_Abstract

Matches event data and runs appropriate method in resource model for re-indexing

Works directly with database for generation of the indexed data. Usually all the data operated via MySQL queries.


What can you use?

Mage_Index_Model_Indexer

getProcessByCode($indexerCode)

getProcessCollection()

processEntityAction($entity, $entityType, $eventType)

Mage_Index_Model_Process

reindexAll()

reindexEverything()

setMode($mode)


What you shouldn’t do…

• Invoke reindexAll method from index model/resource model, because it is better to let admin user know when the index was rebuild.

• Process entity events directly with indexer, instead of passing data through the main controller. You never know which index may depend on this event.


Creating own indexer

• Defining indexer in configuration• Designing index data table• Implementing model • Implementing resource model• Applying index on the frontend


Featured Products

There is easier way to create featured products functionality, but it is a simple example on what should be done for creation own indexer.


Defining index in configuration

<config>



<global>



<index>

<indexer>

<featured_products>

<model>your_module/indexer_featured</model>

</featured_products>

</indexer>

</index>

</global>

</config>

etc/config.xml

Indexer Model

Indexer Code


Designing index data table

• Adding new attribute to catalog product entity called is_featured

• Creating table that will contain product ids of products that are marked as featured products.



$this->addAttribute('catalog_product', 'is_featured', array(

'type' => 'int',

'label' => 'Is featured',

'input' => 'select',

'source' => 'eav/entity_attribute_source_boolean',

'user_defined' => false,

'required' => false

));

Setup Script

Attribute Code

Yes/No Dropdown



$table = new Varien_Db_Ddl_Table();

$table->setName($this->getTable(‘module/featured'));

$table->addColumn('product_id', Varien_Db_Ddl_Table::TYPE_INTEGER, 11, array(

'unsigned' => true,

'nullable' => false,

'primary' => true

));

$this->getConnection()->createTable($table);

Setup Script Table Alias

Table Column



$table = new Varien_Db_Ddl_Table();

$table->setName($this->getTable(‘module/featured'));

$table->addColumn('product_id', Varien_Db_Ddl_Table::TYPE_INTEGER, 11, array(

'unsigned' => true,

'nullable' => false,

'primary' => true

));

$this->getConnection()->createTable($table);

Setup Script Table Alias

Table Column


Implementing Model

class Your_Module_Model_Indexer_Featured extends Mage_Index_Model_Indexer_Abstract

{

protected $_matchedEntities = array(

Mage_Catalog_Model_Product::ENTITY => array(

Mage_Index_Model_Event::TYPE_SAVE,

Mage_Index_Model_Event::TYPE_MASS_ACTION

)

);

}

Defining Matching Events

Entity Type

Event TypeEvent Types


Implementing Model


{

// … other code

protected function _construct()

{

$this->_init(‘your_module/indexer_featured');

}

}

Defining Indexer Resource Model

Resource model


Implementing Model


{

// … other code

public function getName()

{

return Mage::helper(‘your_module')->__('Featured Product');

}

public function getDescription()

{

return Mage::helper(‘‘your_module')->__('Indexes something');

}

}

Defining Indexer Information

Indexer Name in the admin

Indexer Description in the admin


Implementing Model


{

// … other code

protected function _registerEvent(Mage_Index_Model_Event $event)

{

/* @var $entity Mage_Catalog_Model_Product */

$entity = $event->getDataObject();

if ($entity->dataHasChangedFor('is_featured')) {

$event->setData('product_id', $entity->getId());

} elseif ($entity->getAttributesData()) {

$attributeData = $entity->getAttributesData();

if (isset($attributeData['is_featured'])) {

$event->setData('product_ids', $entity->getProductIds());

}

}

}

}

Register Event for Processing

Product Save Registering

Mass Action Registering


Implementing Model


{

// … other code

protected function _processEvent(Mage_Index_Model_Event $event)

{

if ($event->getData('product_id') || $event->getData('product_ids')) {

$this->callEventHandler($event);

}

}

}

Processing Event

Calling processor in resource model

Entity Type Event Type

catalogProductSave($event)

catalogProductMassAction($event)


Implementing Resource Model

class Your_Module_Model_Mysql4_Indexer_Featured extends Mage_Index_Model_Mysql4_Abstract

{

protected function _construct()

{

$this->_setResource(‘your_module');

}

}

Define resource connection

Your module resource prefix




{

// … other code

protected function _reindexEntity($productId = null)

{

$select = $this->_getReadAdapter()->select();

/* @var $attribute Mage_Catalog_Model_Resource_Eav_Attribute */

$attribute = Mage::getSingleton('eav/config')

->getAttribute('catalog_product', 'is_featured');

$select->from($attribute->getBackendTable(), 'entity_id')

->where('value = ?', 1)

->where('attribute_id = ?', $attribute->getId());

Indexing Method

Retrieving only featured product ids



if ($productId !== null) {

if (!is_array($productId)) {

$productId = array($productId);

}

$select->where('entity_id IN(?)', $productId);

$this->_getWriteAdapter()->delete(

$this->getTable(‘your_module/featured'),

array(

'product_id IN(?)' => $productId

)

);

} else {

$this->_getWriteAdapter()->truncate($this->getTable(‘your_module/featured'));

}

Indexing MethodIf it is partial re-index, then delete

only related indexed data

Otherwise clear all indexed data



$sqlStatement = $select->insertIgnoreFromSelect(

$this->getTable(‘your_module/featured'),

array('product_id')

);

$this->_getWriteAdapter()->query($sqlStatement);

}

}

Indexing MethodFulfill index data from select we

created before




{

// … other code

public function reindexAll()

{

$this->_reindexEntity();

}

}

Handling Events

Full index re-build




{

// … other code

public function catalogProductSave($event)

{

$this->_reindexEntity($event->getData('product_id'));

}

public function catalogProductMassAction($event)

{

$this->_reindexEntity($event->getData('product_ids'));

}

}

Reindexing Events

Single Save Product Event

Mass Save Product Event


Applying Index for the frontend

• Observing and event catalog_product_collection_apply_limitations_after– Joining index table to product collection select– Create sub-select filter for collection


Liked it?

Checkout our advanced

training programs:http://www.ecomdev.org/magento-development-training-programs/advanced

Follow our blog posts:http://www.ecomdev.org/blog

http://www.ecomdev.org/magento-development-training-programs/advanced




http://www.ecomdev.org/blog

Questions?

Technology

Magento Indexes