Improving the External Validity of Conjoint Analysis

What is the impact of changing one treatment attribute on patient preferences for that treatment? This is a key question of interest for many patient preference studies. In practice, this is most often done by estimating the causal effect of changing one attribute of a profile while averaging over the distribution of the remaining profile attributes. Formally, this is known as the average marginal component effect or AMCE (see Hainmueller et al 2014). How do researchers do this?

…nearly 90% of the existing conjoint analyses use the uniform distribution. The problem is that the resulting estimate of the AMCE, which we call the uniform AMCE (uAMCE), gives equal weights to all conjoint profiles even when some of them are unrealistic from a substantive point of view. 

Consider the case of doing a conjoint experiment for different cancer treatments. Treatment attributes may include the treatment’s overall survival, progression free survival, adverse event rates, and mode of administration. By using the uAMCE, you are assuming an even distribution across all attribute levels. In practice, however, it is likely that when PFS attribute level is high, OS is high as well; conversely, when PFS is low, OS attribute levels will be low as well. uAMCE does not take into account that a PFS(high)OS(high) or PFS(low)OS(low) profile is much much more likely than a PFS(high)OS(low) or PFS(low)OS(high) attribute profile.

An alternative approach–proposed by de la Cuesta et al. (2021)–is to use a population AMCE (pAMCE). This approach averages over the distribution of profile attributes in a target population of interest, rather than assuming an equal probability for all attribute combinations. Where would the data for this pAMCE come from? The authors argue that it could come from either real world data–such as real-world market shares of different drugs–or a counterfactual distribution that is of theoretical interest.

The authors also propose two new experimental design approaches using pAMCE.

The first approach, which we call design-based confirmatory analysis, incorporates the target profile distribution in the design stage…In the most natural design, which we term joint population randomization, we propose randomizing conjoint profiles according to their target profile distribution rather than the uniform…

Our second approach, model-based exploratory analysis, takes into account the target profile distribution at the analysis stage, after randomizing profiles and collecting data…This approach is useful in estimating the pAMCE when researchers have to randomize profiles based on distributions different from the target profile distribution, such as the uniform. 

The authors provide some useful applications of pAMCE from the world of political science in the full text.

One challenge for implementing pAMCE in life science context is that RWD to populate the profiles may be sparse or there may be limited target attribute profiles. One reason RWD may be limited is that researchers may want to look at patient preferences for new pharmaceutical treatments relative to those already on the market. Since the new treatment may not yet have been used in practice, it is unclear what weight it would receive. Additionally, there may be limited number of treatments available on the market. In this case, pAMCE could collapse to a direct comparison of of 2 or 3 different treatments, rather than a hypothetical exercise of patient preferences across all plausible combinations of treatments of varying attribute levels.

The authors also note that another challenge is that the joint distribution of all attributes may be difficult to obtain in many research applications. To solve this the authors propose using
marginal population randomization design, whereby researchers would randomize each factor independently with its marginal distribution. However, this design requires a stronger assumption that there are no three-way or higher-order interactions between attributes.

Nevertheless, it is useful to understand what implicit assumption one is making when attribute distributions are assumed of equal likelihood (uAMCE) and whether consider the actual attribute distribution in the target population (pAMCE) could improve the external validity of any estimated preferences from a conjoint analysis study.

You can read the full study here.